Method for detecting user gestures from alternative touchpads of a handheld computerized device

ABSTRACT

A method for controlling a control region on a display screen of a computerized system is presented. The method includes obtaining data from a touchpad. The data is associated with a position of a portion of the hand of a user when the user operates the touchpad. The method includes transmitting the data from the touchpad to the computerized device and analyzing the data in accordance with a model of a human hand. In certain embodiments, the method includes detecting an interaction of at least the portion of the hand of the user on the first touchpad with at least one object displayed on the display screen causing at least one property of the object to be controlled in accordance with the interaction of the portion of the hand of the user on the first touchpad.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority, under 35 U.S.C. §119(e), to U.S.Provisional Patent Application No. 61/916,168, titled, “METHOD FOR USERINPUT FROM ALTERNATIVE TOUCHPADS OF A HANDHELD COMPUTERIZED DEVICE,”inventor Tong Luo, filed Dec. 14, 2013. This application is also acontinuation-in-part of U.S. patent application Ser. No. 14/491,671,titled, “METHOD FOR CONTROLLING A CONTROL REGION OF A COMPUTERIZEDDEVICE FROM A TOUCHPAD,” inventor Tong Luo, filed Sep. 19, 2014, whichclaims priority, under 35 USC §119(e), to U.S. Provisional PatentApplication No. 61/880,629, “METHOD FOR USER INPUT FROM ALTERNATIVETOUCHPADS OF A HANDHELD COMPUTERIZED DEVICE”, inventor Tong Luo, filedSep. 20, 2013; this application is also a continuation-in-part of U.S.patent application Ser. No. 14/341,326, “METHOD OF CONTROLLING A VIRTUALKEYBOARD FROM A TOUCHPAD OF A COMPUTERIZED DEVICE,” inventor Tong Luo,filed Jul. 25, 2014, which claims priority, under 35 USC §119(e), fromU.S. Provisional Patent Application No. 61/858,223, entitled “METHOD FORUSER INPUT FROM ALTERNATIVE TOUCHPADS OF A HANDHELD COMPUTERIZEDDEVICE,” inventor Tong Luo, filed Jul. 25, 2013; this application isalso a continuation-in-part of U.S. patent application Ser. No.14/289,260, “METHOD FOR USER INPUT FROM ALTERNATIVE TOUCHPADS OF AHANDHELD COMPUTERIZED DEVICE”, inventor Tong Luo, filed May 28, 2014,which claims priority, under 35 USC §119(e), from U.S. ProvisionalPatent Application No. 61/828,683, entitled “METHOD FOR USER INPUT FROMALTERNATIVE TOUCHPADS OF A HANDHELD COMPUTERIZED DEVICE,” filed on May30, 2013; this application is also a continuation-in-part of U.S. patentapplication Ser. No. 14/284,068, entitled “METHOD USING A FINGER ABOVE ATOUCHPAD DURING A TIME WINDOW FOR CONTROLLING A COMPUTERIZED SYSTEM,”filed on May 21, 2014, which claims priority, under 35 USC §119(e), fromU.S. Provisional Patent Application No. 61/825,621, entitled “METHOD FORUSER INPUT FROM ALTERNATIVE TOUCHPADS OF A HANDHELD COMPUTERIZEDDEVICE,” filed on May 21, 2013; this application is also acontinuation-in-part of U.S. patent application Ser. No. 14/282,331,entitled “METHOD USING FINGER FORCE UPON A TOUCHPAD FOR CONTROLLING ACOMPUTERIZED SYSTEM,” filed on May 20, 2014, which is acontinuation-in-part of U.S. patent application Ser. No. 14/268,926,entitled “METHOD USING A FINGER ABOVE A TOUCHPAD FOR CONTROLLING ACOMPUTERIZED SYSTEM,” filed on May 2, 2014, which claims priority, under35 USC §119(e), from U.S. Provisional Patent Application No. 61/819,615,entitled “METHOD FOR USER INPUT FROM ALTERNATIVE TOUCHPADS OF A HANDHELDCOMPUTERIZED DEVICE,” filed on May 5, 2013; this application is also acontinuation-in-part of U.S. patent application Ser. No. 14/260,195,entitled “METHOD FOR USER INPUT FROM ALTERNATIVE TOUCHPADS OF A HANDHELDCOMPUTERIZED SYSTEM,” filed on Apr. 23, 2014, which claims priority,under 35 USC §119(e), from U.S. Provisional Patent Application No.61/815,058, entitled “METHOD FOR USER INPUT FROM ALTERNATIVE TOUCHPADSOF A HANDHELD COMPUTERIZED DEVICE,” filed on Apr. 23, 2013; thisapplication is also a continuation-in-part of U.S. patent applicationSer. No. 13/770,791, entitled “METHOD FOR USER INPUT FROM ALTERNATIVETOUCHPADS OF A HANDHELD COMPUTERIZED DEVICE,” filed on Feb. 19, 2013;this application is also a continuation-in-part of U.S. Pat. No.8,384,683 B2, entitled “METHOD FOR USER INPUT FROM THE BACK PANEL OF AHANDHELD COMPUTERIZED DEVICE”, filed on May 4, 2010, which claimspriority, under 35 USC §119(e), from U.S. Provisional Patent ApplicationNo. 61/327,102, entitled “METHOD, GRAPHICAL USER INTERFACE, ANDAPPARATUS FOR USER INPUT FROM THE BACK PANEL OF A HANDHELD ELECTRONICDEVICE,” filed on Apr. 23, 2010; the contents of all of all of the abovereferences applications are incorporated herein by reference in theirentirety. U.S. patent application Ser. No. 13/770,791 referenced aboveis also a continuation-in-part of U.S. patent application Ser. No.13/223,836, entitled “DETACHABLE BACK MOUNTED TOUCHPAD FOR A HANDHELDCOMPUTERIZED DEVICE”, filed on Sep. 1, 2011, which is acontinuation-in-part of U.S. Pat. No. 8,384,683 B2, entitled “METHOD FORUSER INPUT FROM THE BACK PANEL OF A HANDHELD COMPUTERIZED DEVICE”, filedMay 4, 2010, the contents of all of which are incorporated herein byreference in their entirety.

BACKGROUND OF THE INVENTION

The present disclosure generally relates to a computerized deviceincluding a touchpad installed on the back panel or other portion of thebody other than the display screen. More particularly, the presentdisclosure relates to a method and graphical user interface that enablesthe user to see the user's finger position and motion from the back orother portion of the device, superimposed on a keyboard layout on thedisplay screen. This makes it easier for a user to input keystrokes andmouse actions from a touchpad that is installed on the back panel orother portion of a handheld device. In an embodiment, the user can alsocontrol and manipulate a virtual keyboard shown in the display screen.Although embodiments of the invention are described with reference to ahandheld computerized device by way of an example, it is understood thatthe invention is not limited by the type of computerized device orsystem.

Handheld computerized devices, i.e., devices including microprocessorsand sophisticated displays) such as cell phones, personal digitalassistants (PDA), game devices, tabletPCs, such as iPad, wearablecomputerized devices, and the like, are playing a more and moreimportant role in everyday life, and are becoming more and moreindispensable. With the advance of technology, and improvements in thehandheld computerized devices' processing power, both function, andmemory space is increasing at an amazing pace. Meanwhile the size of thehandheld computerized devices continues to get smaller and smallermaking the touchpad and display on the device smaller and morechallenging to use.

To meet the challenge of a smaller device display and touchpad, thedesigners of handheld computerized devices typically use two approaches.One approach is to make the keyboard keys smaller and smaller,miniaturizing the keys. Additionally the keyboard keys may be givenmultiple functions—i.e. overloaded, and more complex function keyboardkeys may be introduced as well.

The other approach is to use touch screen keyboards, or so called “softkeys”, on the front panel. Here a user may use a stylus pen or finger toselect the soft keys through a graphical user interface. However due tothe optical illusions introduced by the display screen, and the factthat the user's fingers often are on top of the various display screensoft keys, hence blocking the keys from direct viewing, the soft keysshould not be too small. Another problem is that when the soft keys aretoo small, often a single finger press will activate multiple keys. As aresult, the designer may have to divide the keys into different groupsand hierarchies, and just display a small number of keys at a time onthe screen.

Both current approaches have some drawbacks: the user input area mayoccupy a significant portion of the front panel, and the user inputprocess, although requiring a large amount of user attention to operate,still is very error prone.

Often a user may use one hand to hold the handheld computerized device,and use the other hand to input data, thus, occupying both hands. A userwill often have to go through a long sequence of key strokes, and switchback and forth among different user interface screens, in order tocomplete a fairly simple input. As a result, there is a significantlearning curve for a user to learn the overloaded keys, function keys,key grouping, and key hierarchies in order to operate the handheldcomputerized devices efficiently.

Previous designs including sensors on the back of the device andrepresentations of the user's fingers on the front of the device,however, this work failed to adequately describe a procedure by whichthe indicia of the user's fingers or hands are displayed on the displaypanel.

Systems have been described in which image sensors would obtain an imageof the user's fingers while operating the device, and use this imagedata to better determine which real or virtual keys the user's fingerswere striking Such methods rely, however, on image sensors that arepositioned in such a way as to be capable of viewing the tips of theuser's fingers. This type of image sensor placement is often difficultto implement on many types of handheld user computerized devices.Another drawback of the previous image sensor approach is that it isdifficult to implement in low light situations. This approach may alsobe difficult to implement in situations where there is limited smoothand flat desk or table space.

BRIEF SUMMARY OF THE INVENTION

According to one embodiment of the present invention, a method forcontrolling a control region on a display screen of a computerizedsystem is presented. The method includes obtaining first data from atouchpad. The first data is associated with a position of a portion ofthe hand of a user when the user operates the touchpad. In an example,the first data is not associated with an image of a finger of the userfrom an image sensor. The method then includes transmitting the firstdata from the touchpad to the computerized system. In an example, thetouchpad is located in a location that is different from the location ofthe display screen. In some embodiments, the method further includesanalyzing the first data in accordance with a model of a human hand andassigning the first data to at least one of a plurality of fingers ofthe model. In some embodiments, the method may include detecting, by thecomputerized system, an interaction of at least the portion of the handof the user on the first touchpad with at least one object displayed onthe display screen. The method may then include causing, by thecomputerized system, at least one property of the object to becontrolled in accordance with the interaction of the portion of the handof the user on the first touchpad.

In some embodiments, the method may include detecting a finger swipe ofa first finger of the user on the first touchpad when the user interactswith the at least one object on the display screen. In an example, theobject may correspond to a multipage application displayed on thedisplay screen and the property of the object may correspond to a pagenumber in the multipage application. In an embodiment, the method mayinclude changing the page number of the multipage application inresponse to the finger swipe.

In some embodiments, the method may include detecting a finger swipe ofa plurality of fingers of the user on the first touchpad when the userinteracts with the object on the display screen. In an example, themethod may include changing the page number by a pre-determined numberof pages in response to the finger swipe of the plurality of fingers. Insome examples, the method may include detecting a velocity of the fingerswipe and changing and the page number by the pre-determined number ofpages in response to the velocity. In some examples, the method mayinclude determining a first direction of the finger swipe andincrementing the page number by a pre-determined number of pages inresponse to the first direction. In some examples, the method mayinclude determining a second direction of the finger swipe anddecrementing the page number by a pre-determined number of pages inresponse to the second direction. In an example, the first direction maybe different from the second direction.

In accordance with some embodiments, the method may include detecting aninteraction of at least a second finger of the user with the firstfinger. In some examples, the second finger may be located a secondtouchpad. In some examples, the second touchpad may be located in alocation different from the first touchpad. In some examples, the secondfinger may also be located on the display screen. In some embodiments,the method may include identifying a first position of the first fingeron the first touchpad, detecting a selection of the object, identifyinga second position of the second finger on the second touchpad anddetecting a movement of the first position of the first finger relativeto the second position of the second finger. In some examples, themethod may include detecting the selection of the object by the firstfinger on the first touchpad or the second finger on the secondtouchpad. In some embodiments, the method may include rotating theobject based on the detected movement, altering an axis of rotation ofthe object based on the detected movement, altering the size of theobject based on the detected movement, altering a display characteristicof the object such as color, size, font and the like based on thedetected movement or moving the object based on the detected movement.

In accordance with at least some embodiments, the method may includeidentifying a first position of the first finger on the display screen,detecting a selection of a point of rotation of the object, identifyinga second position of the second finger on the second touchpad anddetecting a movement of the second position of the second fingerrelative to the first position of the first finger. In some examples,the method may include detecting the selection of the point of rotationof the object by the first finger on the first touchpad or the secondfinger on the second touchpad. In some examples, the method may includemoving the object around the point of rotation of the object based onthe detected movement. In some examples, the object may correspond to avirtual joystick and the method may include pushing the handle of thevirtual joystick in a direction and magnitude defined by a correspondingdirection and magnitude of the movement of the second position of thesecond finger relative to the first position of the first finger.

In accordance with at least some embodiments, the method may includedetecting a selection of the object, identifying a first contactposition of the first finger, detecting a change in a characteristic ofthe first contact position and causing the at least one property of theobject to be controlled based on the change in the characteristic. In anembodiment, the method may include detecting a movement of the firstfinger away from the first contact position, detecting a change in theangle of the first finger in the first contact position, detecting anincrease in a touch area of the first contact position and the like. Inan example, the characteristic may comprise at least one of the area,the size or the shape of the first contact position. In an embodiment,the method may include applying a corresponding pressure or a load tothe displayed object based on the change in the characteristic of thefirst contact position.

In accordance with at least some embodiments, the method may includedetecting a movement of the first finger from a first position to asecond position on the first touchpad, repositioning the object in thedisplay screen in accordance with a direction of the movement of thefirst finger or an amount of movement of the first finger from the firstposition to the second position and enabling, for the user, aninteraction with the re-positioned object. In an example, the object maycorrespond to a set of virtual keys in a virtual control region in thedisplay screen.

In accordance with another embodiment, a computer-readable storagemedium is disclosed. The computer-readable storage medium comprisesinstructions to obtain first data from a first touchpad, the first databeing associated with a position of a portion of the hand of a user whenthe user operates the computerized system using the first touchpad, thefirst data not being associated with an image of a finger of the userfrom an image sensor, transmit the first data from the first touchpad tothe computerized device, the first touchpad being located in a locationthat is different from the location of the display screen, analyze thefirst data in accordance with a model of a human hand and assigning thefirst data to at least one of a plurality of fingers of the model,detect an interaction of at least the portion of the hand of the user onthe first touchpad with at least one object displayed on the displayscreen and cause at least one property of the object to be controlled inaccordance with the interaction of the portion of the hand of the useron the first touchpad.

In accordance with another embodiment, a system for controlling acontrol region on a display screen of a computerized system isdisclosed. The system obtains first data from a first touchpad, thefirst data being associated with a position of a portion of the hand ofa user when the user operates the computerized system using the firsttouchpad, the first data not being associated with an image of a fingerof the user from an image sensor, transmits the first data from thefirst touchpad to the computerized device, the first touchpad beinglocated in a location that is different from the location of the displayscreen, analyzes the first data in accordance with a model of a humanhand and assigning the first data to at least one of a plurality offingers of the model, detects an interaction of at least the portion ofthe hand of the user on the first touchpad with at least one objectdisplayed on the display screen and causes at least one property of theobject to be controlled in accordance with the interaction of theportion of the hand of the user on the first touchpad.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a simplified exemplary front panel view of the handheldcomputerized device displaying the position and motion of the user'sfingers holding the back panel, in accordance with one embodiment of thepresent invention.

FIG. 2 depicts a simplified exemplary back panel view of the handheldcomputerized device depicted in FIG. 1, in accordance with oneembodiment of the present invention.

FIG. 3 depicts a simplified exemplary front panel view of the handheldcomputerized device depicted in FIG. 1 displaying a multitude of groupsof keys, in accordance with one embodiment of the present invention.

FIG. 4 depicts a simplified exemplary front panel view of the handheldcomputerized device depicted in FIG. 1 displaying the position andmotion of the fingers holding the back panel and the multitude of groupsof keys depicted in FIG. 3 in the same time, in accordance with oneembodiment of the present invention.

FIG. 5 depicts a simplified exemplary front panel view of a smallerhandheld computerized device displaying the position and motion of atleast one finger in contact with the touchpad of the back panel, inaccordance with one embodiment of the present invention.

FIG. 6A depicts a simplified exemplary front panel view of the smallerhandheld computerized device depicted in FIG. 5 displaying the positionand motion of at least one user's finger in contact with the touchpad ofthe back panel at the touchpad touch points and a multitude of groups ofvirtual keyboard keys similarly depicted in FIG. 3 in the same time, inaccordance with one embodiment of the present invention.

FIG. 6B depicts a simplified exemplary front panel view of a smallerhandheld computerized device displaying the position and motion of onefinger in contact with the touchpad of the back panel, in accordancewith one embodiment of the present invention. The position of a smallvirtual keyboard, composed of a plurality of groups of keys, is alsoshown. Here this virtual keyboard was previously software aligned tocorrespond to the direction of the user's fingers and hand.

FIG. 6C depicts a simplified exemplary front panel view of a smallerhandheld computerized device displaying the position and motion of onefinger in contact with the touchpad of the back panel, in accordancewith one embodiment of the present invention. The position of a smallvirtual keyboard, composed of a plurality of groups of keys, is alsoshown. Here the keys were previously software aligned to correspond tothe direction of the user's fingers and hand, and the spacing betweenthe keys has also been user adjusted by software.

FIG. 6D depicts a simplified exemplary flowchart how biomechanicalmodels of hand and finger movement may be used to display a virtualimage of at least a portion of a hand of a user on a display screen ofthe computerized system of FIG. 8, in accordance with one embodiment ofthe present invention.

FIG. 7 depicts a simplified exemplary front panel view of the handheldcomputerized device displaying another embodiment of the layout ofvirtual keys as the standard virtual keyboard, in accordance with oneembodiment of the present invention.

FIG. 8 depicts a simplified exemplary block diagram of a computerizedsystem capable of executing various embodiments of the invention, inaccordance with one embodiment of the present invention.

FIG. 9 depicts a simplified exemplary flowchart how biomechanical modelsof hand and finger movement may be calibrated and adapted to help turnthe raw touchpad data into an accurate model of the user's hand andfinger positions, in accordance with one embodiment of the presentinvention.

FIG. 10 depicts a simplified exemplary flowchart how predictive typingmethods may be used to improve the accuracy of the appearance of thevirtual hand and fingers while typing, in accordance with one embodimentof the present invention.

FIG. 11 depicts a simplified exemplary flowchart how dynamic changes intouchpad sensitivity may, for finger proximity touchpads, assist inhighlighting the virtual keys about to be struck by a user while typingon the virtual keyboard, in accordance with one embodiment of thepresent invention.

FIG. 12 depicts a simplified exemplary flowchart for generating imagesof the virtual hand and fingers on the device's graphics display screen,in accordance with one embodiment of the present invention.

FIG. 13 depicts a simplified exemplary biomechanical and/or anatomicalmodel of the human hand, showing the internal skeletal structure with askin overlay, in accordance with one embodiment of the presentinvention.

FIG. 14 depicts how the simplified exemplary user's hand or hands may bephotographed by the device's camera or other camera, and this imageinformation may be used to refine the default parameters of thebiomechanical and/or anatomical model of the user's hand, in accordancewith one embodiment of the present invention.

FIG. 15 depicts how an exemplary device camera may be used to obtain apartial image of the user's hand while using the device's touchpad, andthis information also used to update and refine the biomechanical and/oranatomical model of the user's hand, in accordance with one embodimentof the present invention.

FIGS. 16A-16B depict how a simplified exemplary palm angle rotationtransformation may help the system relate raw touchpad data to astandard biomechanical and/or anatomical model of the human hand, inaccordance with one embodiment of the present invention.

FIG. 17 depicts more exemplary details of the relationship between thefinger roots and the hand's overall palm angle, in accordance with oneembodiment of the present invention.

FIG. 18 depicts more exemplary details of the relationship between thehand's palm direction or palm angle and the tips of the user's fingers,in accordance with one embodiment of the present invention.

FIG. 19 depicts how simplified exemplary biomechanical and/or anatomicalmodel data pertaining to the width of the fingers may be used to helpinterpret raw touchpad data, in accordance with one embodiment of thepresent invention.

FIG. 20 depicts how in a more accurate exemplary model, the location ofthe various finger roots will be displaced to some extent from the palmline (which forms the palm angle) by various amounts δri, in accordancewith one embodiment of the present invention.

FIG. 21 depicts how the simplified exemplary system may attempt tocorrelate detected fingertip data from some fingers with finger rootdata from other fingers, determine that some fingertip data is missing,and thus deduce that these fingers are elevated above the touchpad, inaccordance with one embodiment of the present invention.

FIG. 22 depicts how the simplified exemplary system may further assignraw touchpad data to two different hands of the same user, based on theassumption that the range of possible hand angles for the same user islimited by the user's anatomy, in accordance with one embodiment of thepresent invention.

FIG. 23 depicts a first simplified exemplary example of angle basedfinger matching algorithms, in accordance with one embodiment of thepresent invention.

FIG. 24 depicts a second simplified exemplary example of angle basedfinger matching algorithms, in accordance with one embodiment of thepresent invention.

FIG. 25 depicts a simplified exemplary flowchart how biomechanicalmodels of hand and finger movement may be used to display a virtualimage of at least a portion of a hand of a user on a display screen ofthe computerized system of FIG. 8, in accordance with one embodiment ofthe present invention.

FIG. 26 depicts a simplified exemplary flowchart of a “lift and tap”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention.

FIGS. 27A-27F depict a series of simplified exemplary display screenshots of the “lift and tap” technique of key entry depicted in FIG. 26being used to type the first two letters of a “Hello World” message onthe computerized system, in accordance with embodiments of the presentinvention.

FIG. 28 depicts a simplified exemplary flowchart of a “lift and drag”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention.

FIG. 29 depicts a simplified exemplary flowchart of a “lift and tap”technique of key entry modified to use force applied per finger forcontrolling an input from a user to the computerized system, inaccordance with one embodiment of the present invention.

FIG. 30 depicts a simplified exemplary flowchart of a modified “lift andtap” technique of key entry modified to use a third force applied perfinger for controlling an input from a user to the computerized system,in accordance with one embodiment of the present invention.

FIGS. 31A-31B respectively depict simplified exemplary side and topviews of a portion of the touchpad using the contact area resulting froma first force, in accordance with one embodiment of the presentinvention.

FIGS. 32A-32B respectively depict simplified exemplary side and topviews of a portion of the touchpad using the contact area resulting froma second force, in accordance with one embodiment of the presentinvention.

FIG. 33 depicts a simplified exemplary flowchart of a “push and lift”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention.

FIG. 34 depicts a simplified exemplary time-line of the “push and lift”technique depicted in FIG. 33, in accordance with one embodiment of thepresent invention.

FIG. 35 depicts a simplified exemplary flowchart of a “hover and tap”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention.

FIGS. 36A-36F depict a series of simplified exemplary display screenshots of the “hover and tap” technique of key entry depicted in FIG. 35being used to type and enter the numeral “0” on the computerized system,in accordance with embodiments of the present invention.

FIG. 37 depicts a simplified exemplary flowchart of a method 3700 forcontrolling a control region on a display screen of a computerizedsystem, in accordance with an embodiment of the present invention.

FIGS. 38A-38F depict a series of simplified exemplary illustrations,3800, 3802, 3804, 3806, 3808 and 3810 of controlling a control regiondisplayed in a display screen of a handheld computerized device by auser operating a touchpad, in accordance with embodiments of the presentinvention.

FIGS. 39A-38F depict a series of simplified exemplary illustrations,3900, 3902, 3904, 3906, 3908 and 3910 of controlling a control regiondisplayed in a display screen of a handheld computerized device by auser operating a touchpad, in accordance with another embodiment of thepresent invention.

FIG. 40 depicts simplified exemplary illustrations, 4000 and 4002, thatindicate the manner in which a computerized system (e.g., the handheldcomputerized device 100) may interpret a single finger swipe from a useroperating a touchpad on the back side of the computerized device, inaccordance with an embodiment of the present invention.

FIG. 41 depicts simplified exemplary illustrations, 4100 and 4102, thatindicate the manner in which a handheld computerized device mayinterpret a multiple finger swipe from a user operating a touchpad onthe back side of the computerized device, in accordance with anotherembodiment of the present invention.

FIG. 42 depicts exemplary illustrations, 4200, 4202 and 4204 of themanner in which a computerized system (e.g., the hand held computerizedsystem, 100) may detect finger gestures from a user using multiple touchpads and/or the display screen of the computerized system to control anobject displayed on the display screen, in accordance with an embodimentof the present invention.

FIG. 43 depicts exemplary illustrations, 4300 and 4302 of the manner inwhich a computerized system (e.g., the handheld computerized device 100)may detect finger gestures from a user using multiple touch pads and/orthe display screen of the computerized system to control an objectdisplayed on the display screen, in accordance with an embodiment of thepresent invention.

FIG. 44 depicts exemplary illustrations, 4400, 4402 and 4404 of themanner in which a computerized system (e.g., the hand held computerizedsystem, 100) may detect finger gestures from a user using multiple touchpads and/or the display screen of the computerized system to control anobject displayed on the display screen, in accordance with an embodimentof the present invention.

FIG. 45 depicts exemplary illustrations, 4500 and 4502 of the manner inwhich a computerized system (e.g., the hand held computerized device100) may detect finger gestures of a user from multiple touch padsand/or the display screen of the computerized system to control anobject displayed on the display screen, in accordance with an embodimentof the present invention.

FIG. 46 depicts a simplified exemplary flowchart of a method 4600 forcontrolling a control region on a display screen of a computerizedsystem, in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The embodiments of the present invention relate to a handheldcomputerized device including a bit mapped display screen on the frontpanel, and a touchpad installed on the back panel, side panel, or otherarea other than that of the display screen. More particularly, theembodiments of the present invention relate to a method and graphicaluser interface that enable the user to see the user's finger positionand motion from behind the device or other portion of the devicesuperimposed upon a virtual keyboard layout on the front panel.

It is therefore desirable to have a more efficient and user-friendly wayto do user input for handheld computerized devices. The embodiments ofthe present invention present an effective solution for these aboveproblems. The embodiments of the present invention free the originalkeyboard space on the front panel for applications by utilizing thepreviously mostly unused back panel space for user input. Theembodiments of the present invention are able handle both keyboard inputand mouse input. The embodiments of the present invention present astunning graphic user interface on the front panel screen where a usermay see the real-time position and motion of his/her fingers holding theback panel, on top of the display of keyboard layout, which is alsoreferred to as a virtual keyboard. The embodiments of the presentinvention are more precise than current touch screen keyboards byremoving the display layer that presently exists between the fingers andtouch pad. The embodiments of the present invention also move the user'sfingers away from the front panel, so that the user's fingers will notblock the view of the soft key or area that the finger is presentlyoperating on. For smaller handheld devices, such as cell phone, iPhone™or iPad™, the hand that holds the device may now also do input, hencefreeing the other hand for other activities.

Thus an object of the embodiments of the present invention are toprovide a method for a more efficient and user-friendly user input for ahandheld computerized device.

Another object of the embodiments of the present invention are to freeup the space currently occupied by the keyboard on the front panel ofsmall electronic devices, and utilize the mostly unused space on theback panel of the handheld devices for user input purposes.

Another object of the embodiments of the present invention are topresent a visually compelling user-interface design that enables thereal time position and motion of the fingers that hold the device, whichnormally would be hidden from view by the device itself, to be displayedon the front panel as “virtual fingers” together with an optionaldisplay of a virtual keyboard layout. The user's finger positions andkeyboard layout may be displayed either as background image, or as atransparent layer on top of some of all of the applications currentlyrunning on the handheld device. These semi-transparent representationsof the user's finger positions and virtual keyboard allow the user toeasily enter data while, at the same time, continuing to allow the userunimpeded access to the various applications running on the handhelddevice. Thus, for example, applications originally written for acomputer device that had a physical keyboard may be easily run, withoutcode modification, on a tablet computer device that lacks a physicalkeyboard. Thus, these virtual semi-transparent keyboards and methodsthat also give information of finger motion of the user may be highlyuseful.

Another object of the embodiments of the present invention are to enablethe hand that is holding the device to also do user input operations,hence freeing the other hand for other inputs or other purposes.

According to one embodiment, a device and method include a displayscreen on the front panel, which may be a bit-mapped display screen, atouchpad embedded on the back panel capable of sensing the user's fingerpositions and motion, and a graphical user interface. This graphicaluser interface will normally include both software and optional graphicsacceleration hardware to enable complex graphics to be rapidly displayedon the display screen. The device also has an optional virtual keyboardprocessor that displays the keyboard layout, as well as computes anddisplays the user's finger positions on a real-time basis. The user'sfinger position and motion on the touchpad of the back panel may thus becomputed and displayed on the front display screen as a layer, which maybe a semi-transparent layer, on top of all of the other applications.The virtual keyboard processor may also interpret the finger motions,i.e. strokes, and invoke corresponding operations based on the knownlocation of the finger position on the keyboard.

Unlike previous approaches, the user's fingers do not need to beconstrained to fit onto particular regions of the touchpad, but rathermay be disposed in any arbitrary location. Unlike some previousapproaches, although embodiments of the invention may be aided to someextent by real-time video that may provide video information pertainingto at least some portion of the user's hand, visualization of the user'sfingers, in particular the tips of the user's fingers is not necessary.This makes it feasible to use handheld device video cameras designed forgeneral photographic purposes to be used to help in visualizing theuser's hand, without requiring that much of the user's hand in fact bephotographed. There is no requirement at all that the user's fingertipsbe photographed while operating the device.

FIG. 1 depicts a simplified exemplary front panel view of a handheldcomputerized device (100) displaying the position and motion of theuser's fingers (108) holding the back panel, in accordance with oneembodiment of the present invention. The user is holding the handheldelectronic device (100), similar to an Apple iPad™ or equivalent paddevice. The front panel of the device is occupied by a large graphicsdisplay screen (102), which may be a bit-mapped graphics display screen.In some embodiments, the whole front panel screen or front panel may beoccupied by this graphics display screen (102). The user is holding thehandheld computerized device (100) using his or her hands (104), where aportion of the user's thumb (106) is in front of the device over aportion of the front panel, and the user's fingers (108) are behind thedevice. Although device (100) is not transparent, nonetheless thegraphics display screen (102) is shown representing a graphicalrepresentation of the user's fingers (108) as well as regions where theuser's fingers are apparently touching an obscured from view or“invisible” surface at touchpad touch points (110) at the back panel ofthe device. Each of the touchpad touch points (110) may correspond to areal time finger print image of the tip of the user's finger.

FIG. 2 depicts a simplified exemplary back panel view of the handheldcomputerized device (100) depicted in FIG. 1, in accordance with oneembodiment of the present invention. In contrast to the front panel ofdevice (100), previously depicted in FIG. 1, which included a largegraphics display screen, the back panel of the handheld computerizeddevice as depicted in FIG. 2 does not include a large graphics displayscreen, but instead includes a large touchpad (200). As may be seen, theuser's fingers (208) may now be seen positioned above the touchpad withthe tips of the user's fingers (210) touching the touchpad. It isunderstood that the expression “above the touchpad” refers to therelative position of the fingers with respect to the touchpad when thetouchpad is facing upward.

Note that in some embodiments, this back touchpad may be provided as aretrofit or add-on to a handheld computerized device that otherwiselacks such a back touchpad. Such methods and systems, such as “clip on”back touchpads, are described at more length in parent application Ser.No. 13/223,836, the contents of which are incorporated herein byreference in its entirety.

FIG. 3 depicts a simplified exemplary front panel view of the handheldcomputerized device depicted in FIG. 1 displaying a multitude of groupsof keys (300, 302, 304), in accordance with one embodiment of thepresent invention. FIG. 3 depicts one possible optional multitude ofgroups of keys, i.e. a “virtual keyboard,” being displayed on graphicsdisplay screen (102) of device (100). In this example, the “virtualkeyboard” includes a symbol keypad (300), a numeric keypad (302), and aQUERTY keypad (304). Note that in many embodiments, the keys may bedrawn in outline or semi-transparent form so as not to obscure any othergraphical applications running on graphics display screen (102).

The scheme depicted in FIG. 3 allows the user to optionally use atouchpad keypad on the back of the device to input keystrokes and mouseactions, and these inputs will be reflected on the display screen on thefront of the handheld computerized device as “virtual fingers” orequivalent. As previously discussed, this virtual keyboard layoutdisplayed on graphics display screen (102) at the front panel may be astandard or modified QUERTY keyboard or keypad, a numeric keyboard orkeypad, i.e. number entry keyboard, or alternatively some less standardkeyboard or keypad such as a musical keyboard, a Qwerty, Azerty, Dvorak,Colemak, Neo, Turkish, Arabic, Armenian, Greek, Hebrew, Russian,Moldovan, Ukranian, Bulgarian, Devanagari, Thai, Khmer, Tibetan,Chinese, Hangul (Korean), Japanese, or other type of keyboard. Often,this keypad will be a semi-transparent keypad in order to allow the userto continue to view various application programs that are running ondisplay screen (102) below the virtual keyboard.

FIG. 4 depicts a simplified exemplary front panel view of the handheldcomputerized device (100) depicted in FIG. 1 displaying the position andmotion of the user's fingers (108) holding the back panel and themultitude of groups of keys (300, 302, 304) depicted in FIG. 3 in thesame time, in accordance with one embodiment of the present invention.FIG. 4 depicts an example of how a user, typing on a touchpad mounted onthe back of the electronic device, may see a graphical representation ofhis or her fingers (108) displayed on graphics screen (102) of device(100), as well as a display of virtual keyboard layout (300, 302, 304).The user's ability to enter input data to the handheld computerizeddevice (100) is thus enhanced because the user may visually judge thedistances between his or her fingers (108) and the keypad keys ofinterest (300, 302, 304) and move his or her fingers appropriately so asto hit the desired key. The user may also click on hyperlinks, such aslink1, link2, and the like, or other clickable objects or command icons.

Because the user's operating fingers are moved away from the displayscreen, the fingers will not block the view of the display screen's softkeys, soft buttons, links or other areas. These areas on the displayscreen may now be seen more precisely, which in turn allows for moreprecise operation of the device.

The virtual display of the user's fingers may be a valuable feature forsome of the newer tablet computers, such as the Microsoft Surface™series, Windows 8, and the like, which may alternate operating modesbetween a first tablet operating mode designed for traditional touchinput, and a second desktop operating mode, derived from legacy desktopoperating systems, that is optimized for more precise mouse input. Byenabling such tighter control, it becomes more feasible for a user tooperate such “Surface” like devices in legacy desktop mode without theneed to use a mouse or other hand operated pointing instrument.

Because a front keyboard, i.e. mechanically actuated keys, is no longernecessary, the embodiments of the present invention free up the space onthe device that might otherwise have been used for original mechanicalkeyboard space on the front panel, and create room for additional largerdisplays and applications. The embodiments of the present invention makeuse of the presently mostly unused back panel space, thus, enabling thefront display to show substantially larger virtual keys, or virtual keysincluding more space between them that are easier for the user to use.

The embodiments of the present invention may create compelling visualeffects, as well as useful visual effects, because the user may see hisor her fingers (108), which are holding the back panel and thus normallyblocked from view, being virtually displayed on the front panel alongwith a virtual, i.e. computer generated, keyboard layout display (300,302, 304). Because both the user's finger position, finger touch area,each depicted as a circle surrounding a cross, finger motion and thevirtual keyboard are visible from the front panel, the user fingerinputs on the touch panel located on the back panel of the device areboth intuitive and easy to use. There will be no learning curve, and noneed for special training The user input methods of the embodiments ofthe present invention are more precise than traditional touch screenkeyboards because these methods remove the obscuring layer between thefinger and touchpad, and the operating fingers will not block the viewof the display screen. For small handheld devices such as cell phonesand iPhones, the current embodiments of the present invention enable thehand that holds the device to perform text input and other commands,hence freeing the other hand for other activities.

Note that although often a virtual keyboard will be presented,alternative data entry points of interest, such as hyperlinks on aninternet browser, and the like, may also be used according to thesemethods as well.

In one embodiment, the layout of a multitude of groups of virtualkeyboard keys (300, 302, 304), including numbers, letters, and symbolsmay be displayed on an area separated from concurrently running othersoftware applications that are being displayed simultaneously on thescreen of the front panel (much like the traditional separatelydisplayed area often used for soft keys near the bottom of the displayscreen). The virtual keyboard keys (300, 302, 304) may be advantageouslydisplayed in different size or in locations that are not the samelocations that are determined by the other software applications and/orprograms because the virtual keyboard keys (300, 302, 304) may bedisplayed translucently so as to display both the virtual keyboard keys(300, 302, 304) and the underlying concurrently running application orprogram display content.

Devices and systems utilizing the virtual fingers and optional virtualkeyboard embodiments of the present invention advantageously need nothave mechanically actuated and/or permanently dedicated physical QWERTYkeypads or QWERTY keyboards, or any other type of mechanically actuatedand/or permanently dedicated physical keypad or keyboard such as onededicated to number entry. Eliminating mechanically actuated and/orpermanently dedicated keypads or keyboards improves device ergonomics,allow for larger graphics display screens, and also reduces devicecosts.

FIG. 5 depicts a simplified exemplary front panel view of a smallerhandheld computerized device (500) displaying the position and motion ofat least one finger in contact with the touchpad of the back panel, inaccordance with one embodiment of the present invention. Smallerhandheld computerized device (500) may include a cellular phone sizeddevice (e.g. an Apple iPhone™ sized device) including a smaller graphicsdisplay screen (502) virtually displaying the position and motion of amultitude of fingers (108) in contact with the touchpad touch points(110) at the back panel of smaller handheld computerized device (500).

FIG. 6A depicts a simplified exemplary front panel view of the smallerhandheld computerized device (500) depicted in FIG. 5 displaying theposition and motion of at least one user's finger (108) in contact withthe touchpad of the back panel at touchpad touch points (110), and amultitude of groups of virtual keyboard keys (300, 302, 304) similarlydepicted in FIG. 3 in the same time, in accordance with one embodimentof the present invention. FIG. 6 may include similar features as FIG. 4with the exception of using smaller handheld computerized device (500)being held in just one user hand (104), the other hand of the user beingfree to do other tasks.

Adjusting the Virtual Keyboard

Because the virtual keyboard (or keypad) is software generated, it neednot always be presented in the same position. However, in embodiments ofthe invention, at least temporary persistence of the various keys of thevirtual keyboard is desirable, so that the user always knows therelative location of the key that they are going to strike, and so thatthe system can accurately compute the relative distance between theuser's various fingers (as predicted by the anatomical and biomechanicalmodel of the user's hand) and the various keys.

In some embodiments, it may be useful to allow the user to adjust theposition, orientation, and spacing between the virtual keys of thevirtual keyboard either prior to beginning a typing session, or evenduring a typing session. Here, for example, a user may indicate to thesystem by keypress, voice command, virtual key selection, other touchpadinput, etc. that virtual keyboard repositioning is desired. The systemmay then use various options to reposition the virtual keyboard.

In some embodiments, the virtual keyboard will be essentially allowed tofloat on the screen, and the user can then rotate the virtual keyboard,stretch and shrink it, change key spacing, etc., by multi-touch typecommands either on a front display panel touchpad, a back paneltouchpad, or other touchpad device.

In other embodiments, the user may control the position, orientation,and spacing between the virtual keys of the virtual keyboard by verbalcommands such as “rotate right 30 degrees”, or “go up one inch”, and soon.

In some embodiments, the position and orientation of the virtualkeyboard can be set to track the position and orientation of the user'shand(s). In embodiments of the invention, design tradeoffs are takeninto consideration. If the position and orientation of the virtualkeyboard tracks the position and orientation of the user hand(s) tooclosely, then the ability of the software to determine which virtual keythe biomechanical and anatomical model of the user's hand is strikingmay be reduced. Thus, in some embodiments, it may be useful to set thevirtual keyboard generation software to track a time averaged positionand orientation of the user's hand (usually over periods of at leastseveral seconds, or even minutes).

In other embodiments, the virtual keyboard generation software can beset to initialize the position and orientation of the virtual keyboardbased on the position and orientation of the user's hand during aspecified time. This can be as simple as having the user place his orher (hands) on the back touchpad and giving an “initialize keyboardorientation and position” command, either verbally, by pressing a realor virtual key, or by other activation system. Once so initialized, thevirtual keyboard can then maintain its position and orientation untilthe user then decides to reset it.

FIG. 6B depicts a simplified exemplary front panel view of a smallerhandheld computerized device displaying the position and motion of onefinger in contact with the touchpad of the back panel, in accordancewith one embodiment of the present invention. The position of a smallvirtual keyboard, composed of a plurality of groups of keys, is alsoshown. Here this virtual keyboard was previously software aligned tocorrespond to the direction of the user's fingers and hand.

FIG. 6C depicts a simplified exemplary front panel view of a smallerhandheld computerized device displaying the position and motion of onefinger in contact with the touchpad of the back panel, in accordancewith one embodiment of the present invention. The position of a smallvirtual keyboard, composed of a plurality of groups of keys, is alsoshown. Here the keys were previously software aligned to correspond tothe direction of the user's fingers and hand, and the spacing betweenthe keys has also been user adjusted by software.

FIG. 6D depicts a simplified exemplary flowchart how biomechanicalmodels of hand and finger movement may be used to control a virtualkeyboard on a display screen of the computerized system of FIG. 8, inaccordance with one embodiment of the present invention. FIG. 6D depictsthe method includes obtaining data from a touchpad, the data beingassociated with the location and movement of a finger and/or hand of auser when the user operates the computerized system using the touchpad,the data not being associated with an image of the finger of the userfrom an image sensor (620). The method further includes communicatingthe data from the touchpad to the computerized device, the touchpadbeing located in a location that is different from the location of thedisplay screen (622). The method further includes analyzing the data inaccordance with a model of a human hand, and assigning the data to atleast one of a plurality of fingers of the mode (624), computing agraphical representation of at least one finger of the user inaccordance with the model (926), generating a virtual keyboard on thedisplay screen (628), and repositioning the virtual keyboard accordingto either a verbal command from the user or a user input from thetouchpad (630).

FIG. 7 depicts a simplified exemplary front panel view of handheldcomputerized device (100) depicted in FIG. 1 displaying anotherembodiment of the layout of virtual keys (700) as the standard virtualkeyboard, in accordance with one embodiment of the present invention.FIG. 7 depicts another embodiment of the layout of virtual keys (700)may include a modified QWERTY keyboard or keypad that includes splittingthe keyboard in half and displaying each half of the keyboard at anangle adapted for better ergonomic typing than the keyboards depictedpreviously.

In one embodiment, a computer-implemented method includes a handheldcomputerized device, including a screen on the front of the devicecapable of displaying a graphical user interface, and a touch sensitiveback panel or side panel or other area other than the display screen,and a user interface, such as a two dimensional touch sensor. Forsimplicity, this touch sensitive panel, which need not necessarily beflat, and need not necessarily be mounted on the back side of thedevice, hereinafter also referred to as a “touchpad,” “touch sensor,” or“touch sensitive back panel”, but this use is not intended to belimiting.

The touch sensor will determine the motion of the fingers in real time,and the computerized system's or device's software and processor(s) willuse the touch sensor data to compute the real time position and motionof the user's fingers that are touching the touch sensor on the backpanel. These “virtual fingers” will then be displayed on the device'sgraphical user interface on top of a static background where optionallya multitude of groups of keys, including numbers, letters, and symbols(e.g. a virtual keyboard) or hyperlinks may be displayed. By watchingthe motion of the user's virtual fingers on the virtual keyboard, theuser may easily operate the device, and optionally determine preciselywhere to strike a finger in order to hit an intended virtual key.

In one embodiment, the back panel user interface (UI) may be outlined ina distinctive yet non-obstructive color and displayed as a transparentlayer over the current applications; hence all the details of currentapplication and back panel UI are shown to the user at the same time.

In one embodiment, the real time position and motion of the fingersholding the back panel may be displayed on the screen of the frontpanel.

In one embodiment, the layout of a multitude of groups of keys,including numbers, letters, and symbols may be displayed on the screenof front panel as background of real time position and motion of thefingers holding the back panel.

In one embodiment, the real time position and motion of the fingersholding the back panel may be displayed on the static background of amultitude of groups of keys, including numbers, letters, and symbols,enabling the user to precisely strike a finger on an intended key.

In one embodiment, the display of the virtual hand may be creative andartistic. For example, the display may instead show a skeleton, ananimal claw, a furry hand, a tattooed hand, and the like to achieve morecompelling or amusing effects.

In one embodiment, a computer-implemented method, including a handheldcomputerized device includes a touchpad installed on the back panel. Thetouchpad is able to sense the touch point positions, movement, andstroke motion data of a multitude of fingers. The data information ofthe finger motion of one or a multitude of fingers, including the motiontype, e.g., touch, movement, and stroke patterns, and the like, andmotion position, is passed to a virtual keyboard processor, such as acomputer processor. The virtual keyboard processor may analyze thefinger motion, compare the finger positions with the registered positionof the keys, hereinafter referred to as virtual keys, as well as thehyperlinks and other touch buttons of the application program, e.g.,generically the “user entry area”, and then will decide which item inthe user entry area was stroked or actuated. The virtual keyboardprocessor may then invoke the corresponding operation. The virtualkeyboard processor may also update the real time image of the fingers,or finger pads or touch points, or indeed the user hand(s) on the frontscreen after each finger motion.

In one embodiment, the touchpad may be installed on the back panel ofthe handheld computerized device, and may be able to sense the touch,movement, and stroke motion of a multitude of user fingers.

In one embodiment, the information pertaining to the finger motion of amultitude of user fingers, including the motion type, e.g., touch,movement, and stroke action, and the like, as well as motion position,may be passed to a virtual keyboard processor.

In one embodiment, the virtual keyboard processor may analyze the fingermotion, compare finger position with the registered position of thekeys, determine which key was stroked, and invoke the correspondingoperation,

In one embodiment, virtual keyboard processor may update the real timeposition and motion of the fingers holding the back panel.

One embodiment of the present invention includes a graphical userinterface (GUI) for a handheld computerized device. The interface mayinclude a display of a multitude of groups of keys, including numbers,letters, and symbols. The keys may be displayed on a graphical userinterface on the front panel display screen, and indeed this displayarea may occupy the whole screen. Thereby, the content of the graphicuser interface is not blocked by applications, and is shown togetherwith the applications.

One embodiment of the present invention includes a graphical userinterface for a handheld computerized device. This interface includes adisplay of the real time position and motion of the fingers holding theback panel. Here the display is on the front panel screen, and in factmay occupy the whole screen. Due to the advantages of this approach, thecontent of the user's finger position and motion is not blocked byapplications, or by the display of groups of keys, including numbers,letters, and symbols.

One embodiment of the present invention includes a method of assistinguser data entry into a handheld computerized device. This handheldcomputerized device includes at least one touchpad in one embodimentbeing located on a side of the handheld computerized device that isbehind the side of the device that holds the graphics display screen, atleast one graphics display screen, at least one processor, memory, andsoftware. Often, however, the handheld computerized device will lack amechanically actuated and/or permanently dedicated physical QWERTYkeypad or keyboard, and may also lack a mechanically actuated and/orpermanently dedicated physical numeric keypad or keyboard as well. Themethod will usually include displaying at least one data entry locationon the at least one graphics display screen of the device. Often this atleast one data entry location will be a graphical display of a keyboardor keypad that may be included of a multitude of data entry locations.Here, the system will use the touchpad to obtain data on the locationand movement of the user's fingers and/or hand. The system may analyzethe data on the location and movement of the user's fingers and/or handaccording to a biomechanical and/or anatomical model of a human hand,and will assign data on the location and movement of the user's fingersand/or hand to specific fingers on this biomechanical and/or anatomicalmodel of a human hand (usually the user's hand). The system may then usethis biomechanical and/or anatomical model of the human hand to computea graphical representation of at least the user's fingers, andfrequently both the user fingers and the user hand(s). The system willthen display the graphical representation of at least the user's fingers(and again frequently both the user's finger and hand), on the device'sgraphics display screen. Thus the distance between the graphicalrepresentation of the user's virtual fingers on the graphics displayscreen, and the virtual data entry location (such as the virtualkeyboard) will give information that will help the user properlyposition his or her real fingers and/or hand on the touchpad, which inturn will facilitate data entry.

FIG. 8 depicts a simplified exemplary block diagram of a computerizedsystem 800S capable of executing various embodiments of the invention,in accordance with one embodiment of the present invention. Computerizedsystem 800S includes software and hardware that may be used to implementone embodiment of the invention such as a front panel screen (804), aback panel touch pad 800, a virtual keyboard processor (802), anapplication process (806), and a device memory(808). Finger position andmotion data are first collected from back panel touch pad (800), andthen passed to virtual keyboard processor (802). The virtual keyboardprocessor (which will often be implemented by a combination of softwareand hardware such as a microprocessor, graphics processor, touchpadcontroller, and memory) displays the virtual finger position and motiontogether with the keyboard layout on front panel screen (804). Thevirtual keyboard processor also analyzes the finger position and motioninformation data, compares the data with the registered position of thekeys (or hyperlinks) and invokes proper operation in application process(806). The keyboard position information may be programmed in a virtualkeyboard process, or alternatively may be saved in system memory (808).The key-press or hyper-link information that the user intends to relayto the applications may be passed to the virtual keyboard controllereither through memory, or alternatively through inter-processcommunications.

Other Touchpad and Screen Locations.

In one embodiment, the display screen may be located at some distancefrom the touchpad. Indeed, the display screen and the touch pad may noteven be physically connected at all. Rather the touchpad may transmitdata pertaining to the user's hand position to a processor, which inturn may then generate the virtual image of the user's hand and displaythe virtual hand on the display screen, and neither touchpad, processor,or display screen need to be physically connected (although they maybe). For example, data pertaining to the user's hand and finger positionrelative to the touchpad may be transmitted by a wired, wireless, oroptical (e.g. infrared) method to the processor. The processor in turnmay transmit the virtual image of the user's fingers and hand to thedisplay screen by a wired, wireless, or optical (e g infrared)technique. As a result, the user's real hand will be moving close to atouchpad at a different place other than the current display screen. Thedisplay screen may thus be in nearly any location, such as on a regularmonitor, TV screen, projector screen, or on a virtual heads-up eyeglassdisplay worn by the user (e.g. a device similar to Google Glass).

Touch Pads Including Non-Flat Surfaces.

Although touch pads are often flat and roughly rectangular devices,there is no constraint that the touch pads using embodiments of thepresent invention be either flat or rectangular. Indeed in someembodiments, there is advantage to employing touch pads that includevariably shaped and curved surfaces. Such curved and/or variably shapedtouch pads could be then placed on various non-traditional locations,such as on the surface of a ball or cylinder, on the surface of variouscommon devices such as glasses frame stems for virtual heads-up displayssuch as windshields, eyeglasses, and the like, other wearablecomputerized devices such as smart watch bands, steering wheels—eitherfor a vehicle or a game interface, joysticks, and the like, and/or,dashboards, instrument panels, and the like.

Touchpad Technology.

In principle, many different types of touchpad technology may be usedfor this device, including capacitive sensing, conductance sensing,resistive sensing, surface acoustic wave sensing, surface capacitancesensing, projected capacitance sensing, strain gauges, optical imaging,dispersive signal technology, acoustic pulse recognition, pressuresensing and bidirectional screen sensing. However, in a preferredembodiment, touchpad sensing technology that is capable of sensingmultiple finger positions at the same time may be used. Such an abilityto sense multiple finger positions or gestures at the same timehereinafter also referred to as “multitouch” or “multi-touch” sensingtechnology. Touchpads are thus distinguished from previous mechanicalkeyboards or keypads because touchpads are not mechanically actuated,that is, since the surface of a touchpad is substantially rigid andresponds to touch instead of a mechanical deflection, the touchpad givesthe user substantially no indication that the immediate surface of thetouchpad moves where touched, except perhaps for the entire rigidtouchpad moving as a result, even with pressure sensitive touchpadtechnology. Touchpads are further distinguished from previous mechanicalkeyboards or keypads because the shape and/or location of input keys orbuttons on a touchpad are not fixed because the keys and/or buttons areinstead displayed on an electronically controlled screen with theflexibility of software control and not limited by fixed mechanicalelements located on the device.

One example of a multi-touch touchpad embodying the present inventionmay use a touch sensing device commercially available from CypressSemiconductor Corporation, San Jose, Calif. and commonly known as theCypress TrueTouch™ family of products. This family of touchpad productsworks by projective capacitive technology, and is suited for multi-touchapplications. The technology functions by detecting the presence orproximity of a finger to capacitive sensors. Because this touchpadsystem senses finger proximity, rather than finger pressure, it is wellsuited to multi-touch applications because, depending upon the tuning ofthe capacitance detection circuit, various degrees of finger pressure,from light to intense, may be analyzed. Although often used on touchscreens, the projective capacitive technology method may function with abroad range of substrates.

Virtual Finger and hand Position Software (Virtual Keyboard Processor)

As others have noted, one problem with attempting to create “virtualfingers” is that at best, usually just certain regions of the hand, suchas the fingertips and perhaps the palms, may usually be detected byconventional multi-touch sensors. To overcome this issue bootstrappingfrom hand-position estimates has been suggested, which overcomes theinvisibility of structures that link fingertips to palms. Suitablealgorithms could be obtained by using context-dependent segmentation ofthe various proximity image constructs, and by parameterizing the pixelgroups corresponding to each distinguishable surface contact. It wasfound that by path-tracking links across successive images, those groupswhich correspond to the same hand part could be determined, and it waspossible to reliably detect when individual fingers touched down andlifted from the multi-touch pad surface. It has been proposed that anumber of different combinatorial optimization algorithms that usedbiomechanical constraints and anatomical features to associate eachcontact's path with a particular fingertip, thumb, or palm of eitherhand. Such algorithms further operated by assigning contacts to a ringof hand part attractor points, using a squared-distance cost metric, toeffectively sort the contact identities with respect to the ring of handpart attractor points.

A skeletal linked model of the human hand based software that creates abiology-based (biomechanical and/or anatomical) model of joint motionand associated set of constraints has been proposed. The skeletal linkedmodel approach also is based on a software model of the skin that maystretch and bulge in order to accommodate this internal skeleton. Thesoftware models a natural joint axis for four different types of jointsin the human hand, as well as takes into account the relative lengths ofthe underlying hand bone structure, and also accounts for the spaceoccupied by the hand's muscles and skin.

FIG. 25 depicts a simplified exemplary flowchart how biomechanicalmodels of hand and finger movement may be used to display a virtualimage of at least a portion of a hand of a user on a display screen ofthe computerized system of FIG. 8, in accordance with one embodiment ofthe present invention. FIG. 25 depicts the flowchart includes obtainingdata from a touchpad, the data being associated with the location andmovement of a finger and/or hand of the user and not associated with animage of the finger of the user from an image sensor, when the useroperates the computerized system using the touchpad (2510). Theflowchart further includes communicating the data from the touchpad tothe computerized device, the touchpad being located in a location thatis different from the location of the display screen (2520). Theflowchart further includes analyzing the data in accordance with a modelof a human hand, and assigning the data to at least one of a multitudeof fingers of the model (2530), computing a graphical representation ofat least one finger of the user in accordance with the model (2540), anddisplaying the graphical representation on the display screen (2550).

FIG. 9 depicts a simplified exemplary flowchart how biomechanical modelsof hand and finger movement may be calibrated and adapted to help turnthe raw touchpad data into an accurate model of the user's hand andfinger positions, in accordance with one embodiment of the presentinvention. The system may work with adequate accuracy using standardizedmodels of hand and finger relationships. For example, the system mayperform adequately by an initial calibration step where the systeminvites the user to place his or her hand on the display screen, thesystem displays various sized hands, and the user is invited to enter inwhich standardized hand size best fits his or her own hands. The systemmay then use this data for its various calculations. Even more simply,the system may default to an average hand size for initial use, allowingsome degree of functionality to be achieved with no preliminarycalibration.

In one embodiment, it will be useful to better calibrate the system byemploying one or more active calibration steps. These steps may refinethe initial hand model under actual use conditions, and make appropriateadjustment to the various portions of the hand model as will best fitdata that has been obtained under actual use conditions. An example ofthis active calibration process is shown in FIG. 9. Here the system mayinvite the user to do an active calibration step, or alternatively theuser will voluntarily start an active calibration step, in step (900).In one embodiment, the model includes calibration information inaccordance with pressing a portion of the user's hand on the touchpad ina specified manner. To facilitate this calibration step, the system mayoptionally display one or more targets on the screen, which may bekeyboard targets, or alternatively may be specially designed calibrationtargets specifically designed for the active calibration step. Optionalphotographic calibration steps are described for FIG. 14.

In one embodiment referring to FIG. 9, to reduce complexity, the systemmay optionally request that the user calibrate one hand at a time, andindeed may request that the user operate the fingers on his or her handin a manner different from normal typing so as to gather additionaldata. For example, a user may be requested to first extend a specificfinger to a maximum length and press, then to a minimum length andpress, then to the extreme left and press, then to the extreme right andpress and so on, potentially through all fingers and the thumb on a oneat a time basis. It should be apparent that such a data set may thennaturally be translated into a reasonably detailed model of thatparticular user's hand and its capabilities to maintain a number ofdifferent configurations. During the calibration process, the systemwill accumulate touch data by invoking touchpad sensing hardware andcalibration software (902). The system will also make predictions as tothe location of the user's hand and fingers by bootstrapping fromvarious hand position estimates (904). Often the system will track thepositions of the hand and fingers across successive time intervals to dothe predicting, and compute probable finger paths (906). The system willoften use its internal model of the user's hand biomechanical featuresand anatomical features to do the computing, and to help associate thevarious projected paths with the user's fingertips and thumb position,which at least during the active calibration process will be known(908). Here a path is understood to be the line or linkage between atleast one finger root and the associated fingertip or touchpad touchpoint for the associated finger. The system will then refine its modelsof the user's hand biomechanical and/or anatomical features by comparingthe predicted results with real data, and determine if its user handmodel is working with sufficient accuracy in step (910). If it is, thenthis user hand model will then be adopted and used for subsequent uservirtual keyboard data entry purposes (914). If the user hand model isnot working with sufficient accuracy, then the system will attempt toadjust the hand model by varying one or more hand-model parameters(912), and often will then continue the calibration process untilacceptable performance is obtained.

Thus the calibration software enables the biomechanical and/oranatomical model of the human hand to be calibrated more accurately, soas to match the biomechanical and/or anatomical characteristics of aparticular user's fingers and/or hand.

In one embodiment, the realism of the simulated virtual fingers on thescreen may optionally be facilitated by the use of predictive typingmodels. The predictive typing model approach will be particularly usefulwhen the user is typing text on a virtual keyboard, because the systemmay scan the previous text that has been entered, and utilize adictionary and other means, such as the statistical distribution ofletters in the particular language, to make educated guesses as to whatletter is going to be typed next. This educated guess may then be usedto supplement the touchpad data as to last fingertip position andmovement to tend to direct the appearance of the simulated fingertowards the logical next key. Because this system will occasionally tendto guess wrong, however, the user may find it useful to adjust thispredictive typing “hint” to various settings depending upon the user andthe situation. Thus, a user who is an experienced touch typist and whotends to type both fairly quickly and fairly accurately will tend tofind the predictive typing hints useful, because the predictive approachwill tend to work well for this type of user. On the other hand, a userwho is more of a slow and uncertain “hunt and peck” typist may find thepredictive approach to be less useful, and may wish to either reduce thestrength of the hint or potentially even turn the predictive typing“hint” off altogether.

FIG. 10 depicts a simplified exemplary flowchart how predictive typingmethods may be used to improve the accuracy of the appearance of thevirtual hand and fingers while typing, in accordance with one embodimentof the present invention. In a predictive typing system, the softwarewill first access both the biomechanical and/or anatomical model datafor the user's hands (1000), and the latest fingertip and thumb positiondata from the touchpad sensors (1002). The system will then use thisinformation to display the user's virtual hands and fingers on thedevice's display screen (1004). If a predictive typing mode is on(1006), then the system will attempt to deduce (based upon typing speed,as well as the user's consistency in typing speed, and context) what isthe most probable letter or letters that the user is likely to typenext. The system will also attempt to predict the most probable fingeror fingers that the user will use to type this most probable letter(1008). For example, if the user is typing quickly and consistently, andthe context of the word or sentence indicates that a vowel such as “e”is likely, then the system may use this factor in its analysis of thesomewhat noisy finger position data from the touch sensor to increasethe probability that the user's left index finger (often used to type“e” on a keyboard, and which in-fact may not be registering on the touchpad because the user has lifted the left index finger to move to strikethe “e” key), is moving towards the “e” key. When used properly, suchpredictive typing algorithms may help increase the illusion that theuser is looking through the display and onto his or her hands below thedisplay even though the display/computerized device is not actuallytransparent. Conversely, if the predictive typing mode is turned “off”or set to reduced intensity (1010), then the system will not take theprobable next letter into account in its display of the user's hand andfingers and instead just displays using the virtual hand(s) model.

In one embodiment, the efficiency of the predictive typing may befurther enhanced by incorporating the user's history of finger use foreach particular key. For example, one user may have a strong tendency touse the right index finger to type the keys “H” and “J”, and as anotherexample the same user may have a tendency to user his or her left pinkyfinger to type the letter's “A” and “Z”. Here the system may observe theindividual user's typing patterns over time, either as part of aninitial calibration step, or later (and in one embodiment evencontinually) while monitoring the user's typing patterns, and use theuser's individualized finger-to-letter correlation habits as part of thepredictive typing algorithm.

Thus the predictive typing software enables the computerized device tocompute the graphical representation of at least the user's fingers, andoften the user's fingers and hands, with better precision byadditionally using keystroke predictions, in addition to the data on thelocation and movement of the user's fingers and/or hand obtained usingthe touchpad.

In one embodiment, in order to improve the realism of the virtualfingers, additional “finger hover” algorithms may also be used. As usedin this specification, “finger hover” means highlighting or otherwisegraphically altering the appearance of a virtual key on a virtualkeyboard whenever the system believes that the user's finger is eitherhovering above that virtual key, or about to strike that virtual key.For this type of algorithm, use of touchpads that may sense relativefinger proximity to the touchpad surface, such as projective capacitivetechnology touchpads, may be particularly useful.

The sensors and algorithms that detect relative finger-height above asurface may be tuned to various degrees of sensitivity, and indeed thissensitivity level represents an important engineering tradeoff. If thetouchpad is tuned to too high a sensitivity, then it will tend togenerate spurious (false) signals, and also lack precision as toprecisely where on the touchpad a finger is about to land. If the touchpad is tuned to a lower sensitivity, then the touchpad will tend todetect fingertips that are exerting a considerable amount of pressure onthe touchpad surface.

Although many prior art touchpads tend to use a continual or fixed levelof touchpad sensitivity at all times, in one embodiment for the “fingerhover” option described in this specification, use of a dynamic orvariable level of touchpad sensitivity may be advantageous. For example,to detect finger hovering above a key, a touchpad might first operate ata normal level of sensitivity until it detects that a fingertip withinstrategic striking distance of a particular key has left the surface ofthe touchpad. At this point, in order to detect “finger hover” above thekey, the touchpad circuitry might temporarily reset its sensitivity to ahigher level, designed to more precisely detect when the user's fingeris hovering above the key. If the higher level of touchpad sensitivitydetects the fingertip proximity, the key may be highlighted. If thehigher level of touchpad sensitivity does not detect the hoveringfingertip, then the key will not be highlighted. After a short period oftime, about on the order a tenth of a second, the touchpad may then bereset to the normal level of sensitivity to more precisely determine ifthe finger has then actually touched the touchpad, or not.

FIG. 11 depicts a simplified exemplary flowchart how dynamic changes intouchpad sensitivity may, for finger proximity touchpads, assist inhighlighting the virtual keys about to be struck by a user while typingon the virtual keyboard, in accordance with one embodiment of thepresent invention. In other words, FIG. 11 depicts an example of analgorithm to detect and indicate “finger hover”. Here the systemdisplays the virtual keyboard (1100), as well as an overlay of theuser's virtual fingers on or near this virtual keyboard (1102). When thesystem detects that a finger, suspected of being a finger about to pressa key due to the finger's proximity to the key and or predictive typingconsiderations, leaves the touchpad (most likely because the user hasraised the finger above the touchpad in preparation for striking thevirtual key), (1104) the system will momentarily turn the touchpadfinger proximity detector to a higher level of sensitivity (1106), andthe software will look to see if finger hover over the suspected key orkeys may be detected (1108). If the system does not detect that a fingeris suspect of leaving the touchpad, the system returns to step 1102. Ifa finger hover signal may be detected over the suspected key, then thiskey will be highlighted to help guide the user (1110). After a period oftime that will not normally exceed about a tenth of a second or if nofinger hover is detected, the system will once again lower thesensitivity of the finger proximity detector down to the normal level(1112), in order to precisely detect if the finger is about to strikethe key (1114). If the touchpad, now operating at normal sensitivity,now detects that the virtual key has been struck by the user, the systemwill appropriately indicate the keystroke on the virtual key board byfurther graphical changes to the key (1116) and optionally may issue anaudible key-press or key-click sound as well to give further feedback tothe user. Then the system may record the key strike (1118). If theappropriate finger press was not detected at (1114), then the systemrepeats the flow at step (1102).

More generally, the finger hover algorithm approach allows at least onedata entry location (key) to be highlighted on the device's graphicsdisplay screen whenever the computerized device determines that at leastone finger on the user's hand has left the touchpad, and the positionand motion history of the finger is consistent with an ability of thatfinger to strike a position on the touchpad that is consistent with thelocation of the data entry location (key) on the graphics displayscreen.

Graphical Representation of the User's Human Hand(s) and Fingers.

Once the computerized device has obtained data from the touchpad, aswell as any additional predictive typing data, hover detection methoddata, calibration data, and the like, and has updated its internalbiomechanical and/or anatomical model of the user's hand or hands(including the fingers) to reflect this new data, then the system mayutilize this biomechanical and/or anatomical model of the user's hand orhands to compute a graphical representation of at least the user'sfingers, and often the user's hand and figures, suitable for display onthe device's graphics display screen.

A life-like graphical representation of the user's hand and fingers isnot necessary. Often, a more shadow-gram like or cartoon-liketwo-dimensional model (or representation) of the user's hand and fingerswill be all that will be necessary. Often these two-dimensionalrepresentations of the user's hand and fingers need not include much, ifany internal detail. Rather, these representations, may for example,look much like a translucent gray or other colored shadow projection ofthe user's hands and fingers on a surface. Here, the sharpness and thecontrast and the detail of the user's hands and fingers may have reducedsharpness, and have enough distinguishing contrast from other areas ofthe display screen, so as to enable the user to accurately place his orher hands and fingers on the appropriate virtual buttons or virtualkeyboard that is being shown in the graphical display. More fanciful orartistically inspired hand representations are also discussed later inthis specification.

FIG. 12 depicts a simplified exemplary flowchart for generating imagesof the virtual hand and fingers on the device's graphics display screen,in accordance with one embodiment of the present invention. Many ways tographically represent the user's hands and fingers, or at least theuser's fingers, are possible. In one embodiment, based upon thebiomechanical and/or anatomical model of the human hand(s) (1200), andoptionally specific data on the location and movement of the user'sfingers and hand based on the touchpad data (as well as any additionaldata from predictive typing software, or hover detection) athree-dimensional virtual model may be constructed in the device'smemory that depicts the user's hand(s) and fingers (1202).

Based upon the 3D model, a two-dimensional projection of the generaloutlines of the user's hand and fingers may be made upon a mathematicalsurface that corresponds to the surface of the touchpad (1204). Thisprojection may be in the form of a hand and/or finger outline, oralternatively a virtual hand and finger shadow may be produced. Thisprojection may then be combined with the any other data that is beingsent do a memory buffer or graphics display buffer for the displayscreen of the device, and then displayed to the user (1206).

Thus, in one embodiment, the graphical representation of at least theuser's fingers, and often both the user's hand and fingers, on thegraphics display screen may be done by using the previous assignment ofthe data on the location and movement of the user's fingers and/orhand(s) to specific fingers on the biomechanical and/or anatomical modelof the human hand(s) to create a three dimensional model of the user'shand(s) and fingers in the computerized device's memory. Next, atwo-dimensional projection of this three dimensional model of the user'shand(s) and fingers in memory may be made. Here the two-dimensionalprojection may be on an imaginary plane that corresponds in bothdistance and orientation from the model of the user's fingers to thetouchpad. Thus if, for example, the real user's finger is ¼″ above thetouchpad, then the distance between the three dimensional model of theuser's finger and the imaginary plane that corresponds in distance andorientation to the touchpad will also be ¼″. This two-dimensionalprojection on the imaginary “touchpad” plane (virtual touchpad) may beused to generate the graphical representation of at least the user'sfingers on the graphics display screen, and often the user's fingers andhand(s) as well.

Alternatively, in a less computationally intensive scheme, a twodimensional model of the user's hands and fingers may be manipulated tobest fit the previously discussed hand and finger position and motiondata, and this two dimensional model then used for the graphicalrepresentation.

This two dimensional model may be further user selected according to theuser's hand size, and indeed may be calibrated by asking the user toplace his or her hand on the touchpad, thus allowing the system to sensethe dimensions of the user's hand directly.

FIG. 13 depicts a simplified exemplary biomechanical and/or anatomicalmodel of the human hand, showing the internal skeletal structure with askin overlay, in accordance with one embodiment of the presentinvention. This illustration shows the major bones of the hand, with thebones of the index finger and thumb separated in order to allow thejoints to be better visualized. The internal skeletal structure of thehand (1300) is depicted, along with an outline of the skin on the leftside of the hand (1302). The bones of the fingers include the distalphalanges (1304), the intermediate phalanges (1306), the proximalphalanges (1308) and the metacarpals (1310). The thumb lacks theintermediate phalange.

Here the various finger joints include the distal inter-phalangeal joint(dip) (1312), the proximal inter-phalangeal joint (pip) (1314), and themetacarpophalangeal joint (mcp) (1316). The thumb lacks the distalinter-phalangeal joint (dip), and instead includes the interphlangealjoint (ip) (1318) as well as the carpometacarpal (cmc) joint (1320). Inone embodiment for higher accuracy, it may be useful to replace thedefault parameter values of at least the lengths of these various boneswith actual user hand parameters. In general, the closer the variousdefault parameters of the biomechanical and/or anatomical model of thehuman are to the actual user hand parameters, the better. In someembodiments, even the range of joint motion may also be experimentallydetermined, and used to replace one or more joint motion range defaultparameters.

Finger Identifying Algorithms.

In some embodiments, the biomechanical and/or anatomical model of thehuman hand used in the embodiments of the present invention for fingeridentifying algorithms may be based on the following observations.First, the average human hand has four fingers and one thumb. Second, incontrast to the fingers and thumb of the human hand (e.g. FIG. 13 bones(1308), (1306), (1304), which are relatively flexible above themetacarpophalangeal joint (mcp) (1316), the palm of the average humanhand is relatively inflexible below the metacarpophalangeal joint (mcp)(1316). Indeed the positions of the various metacarpophalangeal joints(mcp) (1316) tend to be relatively invariant with respect to rotation ofthe hand. The various metacarpophalangeal joints (mcp) (1316) mayhereinafter also be referred to as the “finger roots”. Finger roots willbe represented by the variable “r”. Alternatively, finger roots may bereferred to as the junction between the finger and the palm.

Third, due to the relatively invariant shape of the palm, theorientation of the user's palm and its angle with respect to other handstructures, such as the relative orientation of the fingers (e.g. middlefinger (1330) is relatively constant. In particular, the orientation orposition of the various “finger roots” (1316) may define a palm linedirection (1332) that will in turn, when the angle of the palm line withrespect to the coordinates of the touchpad are known, help to define thelocation of the various fingers and fingertips.

Fourth, users may generally desire to manipulate symbols using the areaunderneath the uppermost bone of the finger or thumb (1304). Here, thetouch pad data will include various touchpad touch points, identified in(x, y) coordinates in later figures, which will often but not alwayscorrespond to the area underneath the uppermost bone of the user'sfinger and thumb (1304), hereinafter also referred to as the “fingertips”. The touchpad observed location of any given finger or thumb tipwill often be referred to as (x_(i), y_(i)), where x and y are theobserved touchpad data, and “i” refers to or is associated with thefinger that ultimately produced the touch pad touch data.

The raw touch pad data does not include such (x_(i), y_(i)) labels.Instead, the system embodiments may have to make sense of variousincoming touch pad data, attempt to make sense of the data using theunderlying biomechanical and/or anatomical model of the human hand, andthen generate a virtual hand model that is consistent with both thetouchpad data and the underlying biomechanical and/or anatomical handmodel.

It may be simpler to first consider a model of the human hand as itrotates to various positions and touches the touchpad with variousfingers, and determine what sort of mathematical transformations are atwork in generating the raw touchpad data. Once the above is determined,the process may be worked in reverse to generate a virtual model of theuser's hand.

In one embodiment, it is desired to be able to determine which touchpoint belongs to which finger when the system detects multiple touchpoints on the touch pad. Naming convention are described as follows.Thumb=finger 0=F0, index finger=finger 1=F1; middle finger=finger 2=F2;ring finger=finger 3=F3, and little finger (pinky)=finger 4=F4. Aspreviously discussed, the finger number may be represented by thevariable “i”. Thus, when fingers F0 to F4 are present on the touch pad,the problem becomes one of determining the coordinates (x_(i), y_(i)) ofthe fingertips of F0 to F4, and then mapping the coordinates to thebiomechanical and/or anatomical model of the human hand.

Neglecting, for the moment, hand location on the touchpad issues, oneproblem is that users will usually operate the touchpad with the palmline direction of their hands (1322) at an arbitrary angle Θ withrespect to the coordinate system of the touchpad. Thus, an early step tomaking sense of the touchpad data is to determine this angle Θ, and totransform the raw touchpad data by a rotation of angle Θ and see if theraw touchpad data matches up to a sensible biomechanical and/oranatomical model of the human hand. This transformation problem isdepicted in FIG. 16A through FIG. 19.

In one embodiment, a simplified exemplary palm angle (Θ) rotationtransformation may help the system relate raw touchpad data to astandard biomechanical and/or anatomical model of the human hand, inaccordance with one embodiment of the present invention. If the usertouches both the tips of all fingers and thumb and the base or fingerroot of all fingers and thumb onto the touchpad, then the raw touchpaddata would include a series of (x_(i), y_(i)) values for the fingertips, and a series of (x_(ri), y_(ri)) values for the finger roots.

In one embodiment, the system may determine how much the user's hand isrotated relative to the coordinate system of the touch pad using palmangle Θ, then the process of mapping the raw data into the biomechanicaland/or anatomical model of the human hand may be simplified. Thus, adetermination of palm angle Θ is useful.

FIG. 18 depicts more exemplary details of the relationship between thehand's palm direction or palm angle and the tips of the user's fingers,in accordance with one embodiment of the present invention. When usingtouchpads, users will often touch the pad with the fleshy area of thepalm underneath their finger roots (1822). If the finger root touch areainformation is detected by the touch pad, the system may detect thedirection of the palm line of the hands (1322) from the finger roottouch area. Alternatively, the system may use a relaxed finger positiondirection depicted as dashed-dotted line (1825) from touchpad touchpoint (1810) on F1 to touchpad touch point (1820) on F3, or a relaxedfinger position direction from touchpad touch point (1810) on F1 totouchpad touch point (1830) on F4 to approximate the palm line direction(1322) and adjustment angle σ between the relaxed finger positiondirection and the palm line direction, e.g., between line (1825) andpalm line (1322). The system may then determine the angle Θ between thepalm line, and the touch pad coordinates such as the touchpad x-axis.

FIGS. 16A-16B depict how a simplified exemplary palm angle rotationtransformation may help the system relate raw touchpad data to astandard biomechanical and/or anatomical model of the human hand, inaccordance with one embodiment of the present invention. The process ofrotation transforming the raw touchpad data (x_(i), y_(i)) into palmangle corrected touchpad data (x′_(i), y′_(i)) may be done using thefollowing coordinate rotating formula;

$\begin{matrix}{{\begin{bmatrix}x^{\prime} \\y^{\prime}\end{bmatrix} = {\begin{bmatrix}{\cos \; \theta} & {{- \sin}\; \theta} \\{\sin \; \theta} & {{- \cos}\; \theta}\end{bmatrix}\begin{bmatrix}x \\y\end{bmatrix}}}{{{where}\mspace{14mu} x^{\prime}} = {{{x\; \cos \; \theta} - {y\; \sin \; \theta \mspace{14mu} {and}\mspace{14mu} y^{\prime}}} = {{x\; \sin \; \theta} - {y\; \cos \; {\theta.}}}}}} & (1)\end{matrix}$

The system may find the palm angle using the finger root and/or fingertouch points as shown in FIGS. 16A and 18.

The system may also calculate the finger touch coordinates (x_(i),y_(i)), where i=0, 1, 2, 3, 4 for each finger, as well as the palm linerotation angle Θ, and the new coordinates (x′_(i), y′_(i)). Thesecalculations may be done using the formula (1) coordinate rotatingformula shown above. FIG. 16A depicts the before the rotationtransformation (correction) and the before and after results of thisrotation transformation or correction and FIG. 16B depicts the resultsafter the rotation transformation or correction depicting the palm linedirection being substantially parallel to the touchpad's x-axis. It isunderstood that the word substantially herein refers to an accuracysufficient to achieve proper guidance of the virtual finger(s) displayedon the display screen to the extent that the user may be able to guidethe hand to properly strike a virtual key or other control objectdisplayed on the screen and not intending to imply any more accuracythan so required. Thus, more accuracy is required for smaller virtualkeys or control objects than for larger virtual keys or control objectsbut exact anatomical matching to the hand is not required.

In one embodiment, the system may also determine the finger root(x_(ri), y_(ri)) location coordinate for one or more of the fingers F1,F2, F3, F4. Then the system may perform the analysis often based on theassumption that the F1 root coordinate (x_(r1), y_(r1)) is the mostavailable (i.e. most frequently found in the raw touchpad data), whichis often true because the finger 1 finger root commonly touches thetouchpad surface. Alternatively, because the palm does not bend much,the finger F1 root coordinate may be calculated from the other palmtouch points, i.e. other finger roots (x_(ri), y_(ri)).

FIG. 17 depicts more exemplary details of the relationship between thefinger roots (x_(ri), y_(ri)), i.e., roughly finger joint region (1316),and the hand's overall palm angle, in accordance with one embodiment ofthe present invention. Often there will be missing finger root positiondata. Here various assumptions, based on the anatomical characteristicsof the human hand, may be used to fill in the missing data.

For example, the root coordinates for finger 1 are available (x_(r1),y_(r1)), then based on hand anatomy considerations, the position of thefinger 2 root is likely to be, or may be calculated to be:

${x_{r\; 2} = {{x_{r\; 1} + {\frac{1}{2}\left( {w_{1} + w_{2}} \right)\cos \; \theta \mspace{14mu} {and}\mspace{14mu} y_{r\; 2}}} = {y_{r\; 1} - {\frac{1}{2}\left( {w_{1} + w_{2}} \right)\sin \; \theta}}}},$

where w_(i) is the width of finger i and

${\frac{1}{2}\left( {w_{1} + w_{2}} \right)} = L_{12}$

as depicted in FIG. 17.

FIG. 19 depicts how simplified exemplary biomechanical and/or anatomicalmodel data pertaining to the width of the fingers, such as L₁₂, may beused to help interpret raw touchpad data, in accordance with oneembodiment of the present invention. A palm line vertical direction(1930) may be defined running substantially through touchpad touch point(1820) on finger F2 and substantially perpendicular to palm lines(1322). The intersection of palm lines (1322) and palm line verticaldirection (1930) passing through the longitudinal axis of F2 may passthrough the finger root for F2 at (x_(r2), y_(r2)), which may be usedfor the origin of the coordinate rotation axes X, Y. In the same manner,the system may also calculate the likely finger root coordinates forfingers F3 and F4 (in this simplified approximation, the model mayassume that the finger roots are substantially on the same palm line(1322) as per FIG. 16A, 16B, 19, FIG. 13, and elsewhere). The system mayalso calculate the new coordinates for any given finger “i” rootassuming that the hand is rotated at palm angle Θ by also using rotationformula (1). Here, for example, for finger roots i=1, 2, 3, and 4, therotation transformed finger root locations may be expressed as:

$\begin{bmatrix}x_{ri} \\y_{ri}\end{bmatrix}\overset{yields}{}\begin{bmatrix}x_{ri}^{\prime} \\y_{ri}^{\prime}\end{bmatrix}$

FIG. 20 depicts how in a more accurate exemplary model, the location ofthe various finger roots will be displaced to some extent from the palmline (which forms the palm angle) by various amounts δri, in accordancewith one embodiment of the present invention. Referring simultaneouslyto FIG. 17 and FIG. 20, the rotation transformed positions (x′_(ri),y′_(ri)) of the various finger roots after rotation by palm angle Θ maybe calculated by the following process. First, calculate the rotationtransformed finger 1 root position (x′_(r1), y′_(r1)) from (x_(r1),y_(r1)) using formula 1. Second, apply the formulas

$x_{r\; 2}^{\prime} = {{x_{r\; 1}^{\prime} + {\frac{1}{2}\left( {w_{1} + w_{2}} \right)\mspace{14mu} {and}\mspace{14mu} y_{r\; 2}^{\prime}}} = {y_{r\; 1}^{\prime} + \delta_{r\; 2}}}$

Here often, for a still better approximation, the system may assume thatany of finger roots F2, F3 and F4 might be displaced somewhat from thepalm line (1322) by a small amount, represented by δ_(ri) as depicted inFIG. 20. Note that δ_(ri) may be either positive or negative.

Referring simultaneously to FIG. 17 and FIG. 20, similarly for fingersF3 and F4, the system may use the approximation that the finger rootsare usually separated by the width of the fingers to make furtherapproximations or educated guesses such as:

$x_{ri}^{\prime} = {{x_{r{({i - 1})}}^{\prime} + {\frac{1}{2}\left( {w_{({i - 1})} + w_{i}} \right)\mspace{14mu} {and}\mspace{14mu} y_{ri}^{\prime}}} = {y_{r{({i - 1})}}^{\prime} + \delta_{ri}}}$

Similarly, for the thumb (finger F0):

${x_{r\; 0}^{\prime} = {{x_{r\; 1}^{\prime} - {\frac{1}{2}\left( {w_{0} + w_{1}} \right)\mspace{14mu} {and}\mspace{14mu} y_{r\; 0}^{\prime}}} = {y_{r\; 1}^{\prime} + \delta_{r\; 1}}}},$

when

${\frac{1}{2}\left( {w_{0} + w_{1}} \right)} = L_{0\; 1}$

as depicted in FIGS. 19 and 20.

Alternatively, using the various rotation transformed finger rootcoordinates (x′_(ri), y′_(ri)), the system may also perform the inversetransformation using formula (1) to calculate the raw touchpad data rootposition coordinates (x_(ri), y_(ri)) in the original touch padcoordinate system. This later technique is often especially useful fordetermining if any of the raw touchpad data might represent the thumbroot location (x_(r0), y_(r0)). The raw thumb root touchpad data isoften difficult to obtain because sometimes the thumb root does nottouch the surface of the touchpad.

Using the techniques described above, the system may make sense of theraw touchpad data by sorting the set of rotation transformed fingertippositions {(x′_(i), y′_(i))} and finger root positions {(x′_(ri),y′_(ri))} according to ascending (or descending) x value, and thenattempt to pair the rotation transformed possible fingertip data(x′_(i), y′_(i)) with the rotation transformed possible finger root data(x′_(ri), y′_(ri)).

Missing Finger Detection Algorithms.

Often the user will have real fingers elevated far enough above thetouchpad to produce a “missing finger” problem—that is the touchpad rawdata will lack the coordinates of one or more user fingers, and thesystem software may have to attempt to deduce the existence and locationof these one or more “missing fingers”.

In one embodiment, a unique touchID may be assigned for each continuoustouch. Thus when a finger “i” was previously touched to the touchpad andwas lifted later, one may use the touchpad history data obtained by thesystem at earlier time points (usually a fraction of a second earlier,i.e. time (t−i)) to determine the missing finger. Such time data mayalso be used in another alternative approach, to be discussed shortly.For example, at time (t−1) (i.e. the previous history of stored touchpaddata in a time indexed stack of such touchpad data, with fingers F0-F4identified, one has:

{(x_(0,(t−1)), y_(0,(t−1), touchID_(0(t−1))) . . . (x_(4(t−1)),y_(4(t−1), touchID_(4,(t−1)))}

Assume, for example, that currently at time t, the system has a raw setof data for just three touch points from three fingers, such as fingersF0, F1, F2 (although this example is using and numbering fingers F0, F1,F2, other fingers and other finger Fi could be alternatively used). Theraw data would be:

{(x_(t0), y_(t0), touchID_(t0)) . . . , (x_(t1), y_(t1), touchID_(t1)) .. . , (x_(t2), y_(t2), touchID_(t2)) . . . }

If one finds, for example, that touchID_(0,(t−1))=touchID_(t0) andtouchID_(3,(t−1))=touchID_(t1) and touchID_(4,(t−1))=touchID_(t2), thenone may tell from the current data set at time “t” that; (x_(t0), yt₀,touchID_(t0)) belongs to finger 0, and (x_(t1), y_(t1), touchID_(t1))belongs to finger F3, and (x_(t2), y_(t2), touchID_(t2)) belongs tofinger F4. Then one may further determine that fingers F1 and F2 aremissing, i.e. likely elevated from the touchpad. The positions ofvarious combinations of other fingers may also be analyzed by the samemethods.

However at the initial starting time t=0, the history data may not beavailable. Thus, one should determine the missing (e.g. elevated)fingers by one or more various alternate methods, such as the methodsdescribed below.

FIG. 21 depicts how the simplified exemplary system may attempt tocorrelate detected fingertip data from some fingers with finger rootdata from other fingers, determine that some fingertip data is missing,and thus deduce that these fingers are elevated above the touchpad, inaccordance with one embodiment of the present invention. The “missingfingers” include fingers F1 and F2, which are depicted with shading.Missing fingers include a finger that might have been elevated too farabove the touchpad in order to register a touch point imprint on thetouchpad. To cope with missing fingers the system may operate asfollows. First, from the touchpad data set, the system may calculate thepalm angle Θ coordinates for the various fingertips and finger roots.Second, for each fingertip, the system may check if the position of thefingertip is inside of the range of likely finger root j positions usinga formula such as:

${{x_{rj}^{\prime} - \frac{w_{j}}{2}} \leq x_{i}^{\prime} \leq {x_{rj}^{\prime} + \frac{w_{j}}{2}}},$

where w_(i) is the width of finger i.

If the fingertip “i” is within this range j, then the system willattempt to match the fingertip “i” with root j. The system, may forexample, even attempt to match potential fingertip locations from onefinger with the potential finger root data from another finger. Forexample, the system may attempt to match the fingertip data (x′₁, y′₁)with the finger root position (x′_(r3), y′_(r3)).

For the thumb finger F0 root (x′_(r0), y′_(r0)), and pinky finger(finger F4) root (x′_(r4), y′_(r4)), the range may be calculated asfollows:

${{x_{r\; 0}^{\prime} - {length}_{0}} \leq x_{0}^{\prime} \leq {x_{0}^{\prime} + {\frac{w_{0}}{2}\mspace{14mu} {and}\mspace{14mu} x_{r\; 4}^{\prime}} - \frac{w_{4}}{2}} \leq x_{4}^{\prime} \leq {length}_{4}},$

where length₀ and length₄ correspond respectively to L₀ and L₄ and where

$\frac{w_{0}}{2}$

corresponds to L_(B) as depicted in FIG. 21. Note that in this example,the system is also incorporating finger length (i.e. the length betweenthe fingertip (x′_(i), y′_(i)) and the finger root (x′_(ri), y′_(ri)))into its biomechanical and/or anatomical model of the human hand.

In the frequent cases where finger tips may not be successfully matchedwith corresponding finger roots, for each un-matched finger root, thesystem may mark that finger as missing (i.e. likely raised above thesurface of the touchpad, rather than touching the touchpad). See forexample, fingers F1 and F2 in FIG. 21. Here in FIG. 21, the shading offingers 1 and 2 shows that the system has recognized that the fingerstips are missing, probably because the finger tips are elevated asufficient distance above the touchpad.

Missing Finger Move/Display Algorithms.

In order to show a virtual image of the moving finger, along withfingers that are touching the touchpad, in some embodiments it will alsobe useful for the system to maintain a stack of the latest n (where n≧1)sets of finger position history information—i.e. retain a history of themost recent finger positions.

In one embodiment, when, as will frequently be the case, the fingerposition data is insufficient, but the missing finger “i” may beidentified by using the previous algorithm or other methods, one mayapproximate the missing finger's position by assuming that as per anormal biomechanical and/or anatomical hand, the change x and y positionof the missing finger's neighboring fingers (e.g. neighboring change Δxand Δy) will also pertain to any change in location of the missingfinger as well, as described in the following examples.

Assume that the current time is time “t”, and that the locations of thefingers at earlier times (i.e. within a second or a few fractions of asecond) have been saved as frames data such as of t−1 in the stack. Thesystem may compute a weighted mean value such as:

${\Delta \; x_{it}} = \frac{\sum\limits_{j = 1}^{n}\; \left( {{weight}_{j}*\left( {x_{jt} - x_{({j{({t - 1})}}}} \right)} \right.}{\sum\limits_{j = 1}^{n}\; {weight}_{j}}$and${{\Delta \; y_{jt}} = \frac{\sum\limits_{j = 1}^{n}\; {{weight}_{j}*\left( \left( {y_{jt} - y_{j{({t - 1})}}} \right) \right.}}{\sum\limits_{J = 1}^{n}\; {Weight}_{j}}},$

where j=[1, n] are the touching fingers.

Using the above scheme, then the current position for the missing finger“i” may be calculated as follows.

i x_(it) =x _(i(t−1)) +Δx _(it) and y _(it) =y _(i(t−1)) +Δy _(it)

Note that for (Δx_(it), Δy_(it)) calculations, one may also use othermathematical methods such as arithmetic mean, median geometrical mean,harmonic mean, and so on.

Finger to Hand Matching Algorithms.

Often, particularly for larger devices, the user may operate the deviceusing two hands. When the user is operating the touch pad with two handsat the same time, the system additionally should be able to decide towhich hand the touch points belong to in order to show the user's twohands properly in the display. The system may use various algorithms tohelp with this decision.

In one embodiment, the system may use the range information on thecoordinates after rotating the data by palm angle Θ, as is shown on FIG.21. In this example, all touch points within the following range may beassumed (i.e. mapped into) one hand. The criteria here may be:

For x [4_(r0)−length₀, x′_(r4)+length₄] and for y: [0, length₂] or[0,max{length₀ . . . length₄}]

The system may also use the touch angle information for touch points andpalm line angles to help assign the raw touchpad data to one hand or theother. Here, for example, the system may assume that both hands belongto the same individual, and essentially extend the biomechanical and/oranatomical model of the human hand to also put in some simplified humananatomical constraints regarding the relationships between the angles ofone hand and the angles of the other hand.

FIG. 22 depicts how the simplified exemplary system may further assignraw touchpad data to two different hands (left hand (2202) including F0Lthrough F4L, and right hand (2204) including F0R through F4R) of thesame user, based on the assumption that the range of possible handangles for the same user is limited by the user's anatomy, in accordancewith one embodiment of the present invention. According to thepreviously discussed multi-touch protocol, the touch angle of a touchpoint may also be determined along the long touch side defined asfollows. That is, usually a finger will touch in a roughly oval patternwith the long axis of the oval, i.e. the long touch side, correspondingto the touch angle α of a touch point. For example, based on humananatomical considerations, the angle α between the touch pointdirections D4L, D2R and the associated respective palm line verticaldirection (2220, 2230) will generally be in the range of [0, 90]degrees. The palm line vertical direction (2220, 2230) is substantiallyperpendicular to associated palm lines left and right (1322, 2222)respectively. In this example, palm line vertical direction (2220) maybe associated with finger F4L through touch point (2224) and palm linevertical direction (2230) may be associated with finger F2R throughtouch point (2234)

Alternatively or additionally, in one embodiment, the system may alsopartition the touchpad area (for example, split the touchpad area into aleft half and a right half) and assign some or all of the touch pad datafrom the left half to the user's left hand, and assign some of all ofthe touch pad data from the right side of the touchpad to the user'sright hand.

Angle Based Finger Matching Algorithms:

FIG. 23 depicts a first simplified exemplary example of angle basedfinger matching algorithms, in accordance with one embodiment of thepresent invention. Angle based methods may be used to match raw touchpaddata with specific user fingers. These alternative, angle-based, fingermatching algorithms may be implemented as follows. First, perform or doa best fit between the touchpad data and the biomechanical and/oranatomical model of the user's hand, and second, use this best fitbiomechanical and/or anatomical model of the user's hand, to find apoint substantially along a mid-line (2310) of middle finger F2. Here,for example, one may have mid-line of middle finger F2 pass through thecenter of the palm, e.g. palm center point, (x_(c), y_(c)) or otherpoint on the palm (any palm center point (x_(center), y_(center)) may beused so long as it is inside a region bounded by the fivemetacarpophalangeal joints (x_(ri), y_(ri))). The coordinates of thepalm center point may be calculated based on the finger model, and knownfinger positions.

Continuing the above algorithm with the second step, find (e.g.calculate) the finger root (metacarpophalangeal joint) coordinates(x_(r0), y_(r0)) . . . (x_(r4), y_(r4)) and calculate the angle α_(ri)of finger root “i” to palm center:

α_(ri) =a tan 2(x _(ri) −x _(center) , y _(i) −y _(center)).

The angle calculated by a tan 2 has a range within about −π to +π.Third, sort the α_(ro), α_(ri) . . . α_(r4) in ascending or descendingorder.

FIG. 24 depicts a second simplified exemplary example of angle basedfinger matching algorithms, in accordance with one embodiment of thepresent invention. Continuing the above algorithm with the fourth step,find all fingertip coordinates (x₀, y₀), (x₁, y₁) . . . (x₄, y₄) andcalculate the fingertip angle α_(i) to palm center (x_(C),y_(C))=(x_(center), y_(center)), where α_(i)=a tan 2(x_(i)−x_(center),y_(i)−y_(center)). Next, sort α_(i) in the same order as the α_(ri).Then, match the corresponding finger to the associated angle, as perFIG. 24. The advantage of this approach is that one does not need toperform coordinate rotation to match the fingers. Instead, the a tan 2calculations may be done by computationally faster methods, even bytable lookup methods, as needed.

Iterative or Best Fit Embodiments

In some embodiments, particularly when the raw touchpad data iscluttered or otherwise confusing, the hand and finger analysis softwarediscussed above may operate by an iterative process. For example, thesoftware may make tentative assignments between the raw touchpad dataand one possible set of fingertip, finger root, or palm touch points onthe previously discussed biomechanical and/or anatomical model of thehuman hand (here the user's hand), and score the results according tohow close the raw touchpad data, either before or after various possibletransformations, may fit with a known hand configuration. The softwaremay then explore other possible hand configurations and transformations,and then select or choose the hand configuration and/or transformation(e.g. rotations, translocations, missing fingers, and the like) thatproduces the highest overall score. The software will then use thehighest scoring hand configuration and orientation model for virtualhand display purposes.

Optional Imaging.

In some embodiments, to improve accuracy (that is to replace standardhuman hand biomechanical and/or anatomical model default parameters withactual user calibration parameters), it will be useful to acquire animage of the user's hands, and to employ various image processing andanalysis techniques to analyze this image of the user's one or morehands to better estimate the relative length of the various bones of theuser's hands. Indeed, in the event that the user has lost one or morefingers, the system may then use this information to make correspondingchanges in its biomechanical and/or anatomical model of the human hand.In other words, the model may include calibration information associatedwith an image of at least a portion of the hand of the user.

FIG. 14 depicts how the simplified exemplary user's hand or hands may bephotographed by the device's camera or other camera, and this imageinformation may be used to refine the default parameters of thebiomechanical and/or anatomical model of the user's hand, in accordancewith one embodiment of the present invention. In acquiring such images,often it is useful to have the system provide a standardized background,such as a series of distance markings, grid, graph paper, and the like(1400) in order to better calibrate the image of the hand and correctfor image distortions. This standardized background may additionallyinclude various color, shades of gray, and resolution test targets aswell. The background may be conveniently provided by, for example,electronically providing one or more background image sheets (e.g. ajpeg, png, pdf or other image file) for printing on the user's printer.

In one embodiment, the user may put each hand on background (1400), andtake a photo of the hand(s) (1402) with either the computerized device'scamera or other camera. This image may then be analyzed, preferably byan image analysis program. The background image will help correct forany image distortions caused by different camera angles, and the like.The user hand image analysis may be done onboard the user's handheldcomputerized device, but it need not be. In an alternative embodiment,the user may upload one or more images of the hand taken by any imagingdevice to an external image analyzer, such as a remote internet server.In either event, the image analyzer will analyze the user's skin or handoutline appearance (1404), deduce the most probable lengths one or morebones of the user's hand, such as the user's various finger and thumbbones, and send this data or other data to correct the defaultbiomechanical and/or anatomical model of the user's hand(s) back to theuser's computerized device, such as for example during calibration step906 referenced in FIG. 9 above.

Alternatively, at least with more sophisticated and possiblynext-generation touchpads capable of providing position details for alarge number of contact points, the user may calibration the touchpad byfirmly pressing a portion or all of the user's hand on the touchpad, andallowing a highly capable touchpad to in turn precisely render theresulting handprint. A compute program may then analyze thetouchpad-derived handprint, extract parameters such as finger jointpositions, probable finger and hand bone lengths, and the like andderive the same information as previously discussed for the photographiccalibration step above. In other words, the model includes calibrationinformation in accordance with pressing a portion of the hand of theuser on the touchpad.

Alternatives or Supplements to the Touchpad.

In an alternative embodiment, information on the user's finger placementmay be obtained using optical methods. Thus in an alternativeembodiment, the touchpad sensor may be an optical method such as one ormore cameras. These camera(s) may keep track of the user's hand andfinger positions, and this data may then be fed into the biomechanicaland/or anatomical model of the human hand(s) to compute a graphicalrepresentation of at least the user's fingers as described previously.

Real Time Video Updating

In another embodiment, image information may also be used to refine thebiomechanical and/or anatomical model of the user(s) hands in real timewhile the user is using the touchpad.

FIG. 15 depicts how an exemplary device camera (1500) may be used toobtain a partial image of the user's hand (1506) while using thedevice's touchpad (1508), and this information also used to update andrefine the biomechanical and/or anatomical model of the user's hand, inaccordance with one embodiment of the present invention. The rearmounted device camera (1500), which often will have a very limited fieldof view at close range (1502), may nonetheless be used to obtain a realtime video image of a portion or part (1504) of the user's hand (1506)while the user is using a rear mounted touch pad (1508) using a touchpad mounted on the back of the computerized device (1510). At the sametime, touch pad data gives the position of the user's index finger(1512) as a strong touch pad signal, and the position of the user'smiddle finger (1514) as a weaker touch pad signal (1514).

Note that although the portion of the hand (1504) that may be directlyvisualized by video camera (1500) does not include any image informationat all pertaining to the position of the user's fingers, the imageinformation (1504) does provide a useful series of further constraintsupon the biomechanical and/or anatomical model of the user's hands.Thus, the partial hand image information, in conjunction with the touchpad data (1512), (1514), and optionally with a refined biomechanicaland/or anatomical model of this user's hand (if available) obtained inFIG. 14, above, may improve the accuracy of the depiction of the user'shand and fingers.

In some embodiments, for amusement or artistic purposes, the user maynot wish to have a fully accurate anatomical model of the user's virtualhand displayed on the screen, but may instead prefer a variant, such asa realistic depiction of a “monster hand” with fingers being replaced byclaws, fur, or pads, and the like, or of a skeleton hand that shows theunderlying biomechanical and/or anatomical estimation of the user's handbones as per FIG. 13.

In one embodiment, the system software may also be configured to renderthe user's fingers and hands as various hand variants when displayed.Generally, these hand variants will still provide realistic informationpertaining to the user's hand and finger placement, but will alsoprovide this information as various user artistic options that often maybe customized according to user preference.

Three Dimensional Multi-Touch Gesture Controls.

Commonly, touchpad controls to a computerized system have focused on twodimensional finger gesture controls requiring finger contact on thelocally two-dimensional touchpad surface, even if that surface as awhole may be curved or otherwise project into the third dimension tosome extent. In contrast, the embodiments of the present invention,which may operate using a biomechanical and anatomical model of thehuman hand, may include a three dimensional gesture component thatenables various types of three dimensional multi-touch gesture controlsdescribed below. Three dimensional multi-touch gesture controls may beadvantageous in applications where the user needs to touch a portion ofthe touchpad continually, such as for example, when the user holds ahandheld computerized device including a touchpad on the backside of thedevice. The three dimensional multi-touch gesture controls may help thecomputerized system differentiate touches on touchpad control regionsintended as control inputs from touchpad touches used to merely hold thedevice.

In some embodiments, the three dimensional sensing aspects of thepresent invention may be used to control virtual keyboard data entry toa computerized system by various “lift and tap”, or “lift and drag”, or“lift and other gesture” type modes for data input. More complexvariants can also implement other commands, such as “lift and tap androtate, e.g. with two fingers”, “lift and tap, and enlarge, e.g. withtwo fingers”, and so on.

In one embodiment, the biomechanical and anatomical model of the user'shand may inform the system when one or more user fingers are positionedon the touchpad so as to be above a corresponding control region of thetouchpad, such as above a key of a virtual keypad, virtual keyboard, orabove a hyperlink, and/or the like, but not yet touching thecorresponding control region. Because the model of the hand accuratelydetermines the location of the one or more user fingers even when theuser's finger is not touching the surface of the touchpad, the“off-touchpad” finger location may be used for three-dimensional gesturecontrol.

In one embodiment, the control region of the computerized system may beon a touchpad including an integrated display screen located insubstantially the same location. For example, integrated touchpads mayinclude both display screen and touchpad built in layers and accessiblefrom the same surface or side of the computerized device and thuslocated substantially in the same location even though the layers may beseparated by small dimensions relative to the touchpad surface length orwidth. In an alternative embodiment, the control region of thecomputerized system may be on a separate and/or additional touchpadbeing located in a location that is different from the location of thedisplay screen as previously described in reference to FIG. 8. Forexample, integrated touchpads may include both display screen andtouchpad built in layers and accessible from the same surface or side ofthe computerized device and thus located substantially in the samelocation even though the layers may be separated by small dimensionsrelative to the touchpad surface length or width.

In one embodiment, the user moves a finger onto a control region of thetouchpad, and this finger is in contact with the touchpad. Thecomputerized system may determine if the user wants to activate thatcontrol region, e.g. press the virtual key or control region to generatean input to the computerized system using, for example, a “lift and tap”type control scheme described as follows. When the system does notreceive inputs according to the “lift and tap” or “lift and relatedtype” control schemes described below, the initial finger touch, even iftouching a control region of the touchpad, may not generate unwantedcontrol inputs enabling the user to continue just safely holding thetouchpad.

FIG. 26 depicts a simplified exemplary flowchart of a “lift and tap”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention. FIG. 26 depicts the “lift and tap” technique includesobtaining (2510) data from a touchpad, the data being associated withthe location and movement of a finger and/or hand of the user and notassociated with an image of the finger of the user from an image sensor,when the user operates the computerized system using the touchpad. The“lift and tap” technique further includes communicating (2620) the datafrom the touchpad to the computerized device and analyzing (2630) thedata in accordance with a model of a human hand, such as referenced inFIG. 9-FIG. 25.

FIGS. 27A-27F depict a series of simplified exemplary display screenshots of the “lift and tap” technique of key entry depicted in FIG. 26being used to type the first two letters of a “Hello World” message onthe computerized system, in accordance with embodiments of the presentinvention. FIGS. 27A-27F were obtained as screen shots or grabs in thatrespective order from a few seconds of video showing the display screenof a prototype handheld computerized device while the user proceeded totype at touch-typing speed using the “lift and tap” technique referencedin FIG. 26. The system has already assigned the touch data from thetouchpad to at least one of the multitude of fingers of the model,computed a graphical representation of the at least one finger of theuser, in accordance with the model, and now displays the graphicalrepresentation of the fingers (F1, F2, F3) and hand (2701) of the useron a display screen (2700) of the computerized system. Note that hand(2701) including fingers (F1, F2, F3) is displayed clearly as a virtual,i.e. computer-generated, hand because the palm includes square edges,the fingers include straight sides, and the joints between the fingersand the palm are not continuous.

Referring simultaneously to FIG. 26, FIG. 27A, and FIG. 27D, in oneembodiment, the system (optionally) determines (2640), in accordancewith the touchpad data and the model of the human hand, that at leastone user finger (F1, F2, F3) is initially touching, or is in contactwith, the region of the touchpad corresponding to a virtual key (2702,2703, 2705, 2725) or other control region.

In one embodiment, when the system first detects that a particular userfinger initially touches or is in contact with a virtual key or othercontrol region, the system may optionally generate a graphicalrepresentation associated with the control region being touched ondisplay screen (2700) of the computerized system. For example, FIGS.27A-27F further depict the system is generating and displaying ondisplay screen (2700) a graphical representation of a virtual keyboardincluding a multitude of virtual keys, including virtual keys (2702,2703, 2705, 2725), corresponding to control regions on the touchpad.

In one embodiment, the system may then change the appearance of thegraphical representation of the touched virtual key or other controlregion to show or indicate the control region is being initial touchedthus providing confirmative feedback to the user. For example, thechange of the display image of the touched virtual key may be shown as achange of size, color, shape, slight displacement of the position of thecontrol region image, slight change in display type such as distortionof the control region image, flashing, and/or the like. For example,FIG. 27A depicts virtual keys (2702, 2703, 2705), which are beinginitially touched by respective user fingers (F1, F2, F3) aretemporarily displayed including a slightly larger size and a slightlybolder upper border than the remaining untouched keys.

Referring simultaneously to FIG. 26, FIG. 27B, and FIG. 27E, in oneembodiment, the user may next lift the at least one user finger, e.g.graphically represented by finger F2 in FIG. 27B, and finger F3 in FIG.27E, and the system determines if the now missing in touch contact orlifted finger is most likely positioned above the same previouslytouched virtual key or other control region on the touchpad. In someembodiments, the system will use the biomechanical and anatomical modelof the user's hand to make the above determination. In other words, thesystem determines (2650), using the model that at least one finger ofthe user, e.g. graphically represented by finger F2 in FIG. 27B, andfinger F3 in FIG. 27E, is positioned above but not touching the controlregion of the touchpad, e.g. graphically represented by virtual key “H”in FIG. 27B, and virtual key “E” in FIG. 27E. The model may be used todetermine, for example, that although the touchpad may no longerdirectly sense that the user's finger is in contact with that particularvirtual key or other control region, nonetheless the finger ispositioned directly above the control region, in accordance with touchdata from other regions of the user's hand and/or fingers and theconstraints of the model of the human hand.

It is understood that even if the user initially positions his or herhand and associated fingers in contact with the touchpad such that theat least one finger, e.g. graphically represented by finger F2 in FIG.27B, and finger F3 in FIG. 27E, does not initially contact the controlregion, then the system may still properly determine what control regionis positioned directly below the at least one finger not in contact withthe control region. In other words, step 2640 described earlier may bean optional step in some embodiments because the model may determine thefingertip locations even with the at least one finger not initially incontact with the touchpad but hovering over the control region using theconstraints of model of the human hand.

Still referring simultaneously to FIG. 26, FIG. 27B, and FIG. 27E, inone embodiment, the system may temporarily—for example, a first timeinterval between 0.05 and 5 or even 10 seconds, change the appearance ofthe graphical representation of the virtual key or other control regionthe at least one finger is hovering over. In other words, when the atleast one finger is positioned above but not touching the first controlregion, the system may display the graphical representation of thecontrol region with a different appearance than the prior appearance ofthe control region. The difference in the graphical representation maybe an enlargement of the represented virtual key or other controlregion, as shown at (2720) and (2730), or the difference may be anothervisual change such as a change of size, color, shape, slightdisplacement of the position of the control region image, slight changein display type such as distortion of the control region image,flashing, and/or the like.

In another embodiment that might be useful for handicapped individuals,the system may instead produce or generate a sound signal audible to theuser instead of, or in addition to, the changed appearance of thegraphical representation of the control region when the at least onefinger of the user is positioned above but not touching the controlregion. Note that in FIG. 27B the graphical representation of virtualkeys (2702, 2705) remains displayed as in FIG. 27A because user fingers(F1, F3) continue to touch the touchpad in FIG. 27B.

Referring simultaneously to FIG. 26, FIG. 27C, and FIG. 27F, in oneembodiment, when the user lowers the at least one finger (e.g.graphically represented by F2 in FIG. 27C, and F3 in FIG. 27F) back ontoor touching the touchpad in that region of the touchpad that correspondsto the control region (e.g. graphically represented by virtual key “H”in FIG. 27C, and virtual key “E” in FIG. 27F). In one embodiment, toprevent false inputs and/or to enable finger re-positioning withoutcommand input, the user's finger lowering may optionally be required tohappen within a certain time interval or “first period of time” (usuallyon the order of between 0.01 seconds and 5 or even 10 seconds after thesystem optionally changes the appearance of that particular key or othercontrol region at the start of step 2650.

In one embodiment, the system may then verify, using the biomechanicaland anatomical model of the user's hand, that the user's at least onefinger has now “struck” or “tapped” the particular virtual key or othercontrol region. In other words, the system determines (2660) that the atleast one finger is subsequently touching the control region inaccordance with the data and the model. In one embodiment, the systemmay record, or register that the appropriate virtual key has beenpressed by storing a record of that action in memory. In FIGS. 27A-27Cthe user is inputting a command to the computerized system to type theletter “H”. The system recognizes by the lift and tap action of userfinger F2 that the user is commanding the system to type the letter “H”,and the system generates and displays a corresponding letter “H” (2722)on display screen (2700) to confirm the execution of the command. InFIGS. 27D-27F the user is inputting a command to the computerized systemto type the letter “E”. The system recognizes by the lift and tap actionof user finger F3 that the user intends to command the system to typethe letter “E”, and the system generates and displays a correspondingletter “E” (2732) to confirm the execution of the command.

In one embodiment, the system may optionally change the displayedappearance of the graphical representation of the struck or tappedvirtual key or other control region, often back to either its originalappearance (as depicted in FIG. 27C and FIG. 27F) or an optionaldifferent altered appearance (e.g. a “key struck” appearance) tovisually confirm the control region is touched or struck. In oneembodiment, the altered appearance may include visual change such as achange of size, color, shape, slight displacement of the position of thecontrol region image, slight change in display type such as distortionof the control region image, flashing, and/or the like. In oneembodiment, the optional different altered appearance may be displayedfor a short second period of time or time interval, often in the 0.05 to1 second range, but may extend longer, such as up to 10 seconds. In oneembodiment, alternatively or additionally, the system may also generatean auditory signal that the user's actions have resulted in the pressingof a particular virtual key or other control region.

FIG. 28 depicts a simplified exemplary flowchart of a “lift and drag”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention. The above embodiments may be extended to other inputtechniques, such as “lift and drag”. In a “lift and drag” technique, inone embodiment, the control region may include an elongated controlregion with a length substantially greater than the longitudinal lengthof the surface region of the user's at least one finger when contactingthe touchpad. Such an elongated control region may include a virtualslider (e.g. a virtual linear control region), virtual rectangular orcircular rotation (e.g. a virtual control knob) or virtualexpansion-contraction control region, and the like. In anotherembodiment, the control region may include a multitude of file names,images, icons, and the like, so that the user may drag these file names,images, icons and the like via the drag technique to, for example,execute a move and/or copy command on the corresponding files or virtualobjects.

In one embodiment, after determining that the at least one finger issubsequently touching (2660) the first control region in accordance withthe data and the model and optionally within a certain third timeinterval (usually on the order of between 0.0 seconds and 5 or even 10seconds after the user's finger initially contacts the touchpad), theuser may move or slide the at least one finger on the elongated controlregion of the touchpad. The system may then verify, using thebiomechanical and anatomical model of the user's hand, that the user ismoving or sliding the at least one finger over the elongated virtual keyor other control region. In other words, the system determines (2810)that the at least one finger is subsequently touching the first controlregion in accordance with the data and the model. The system may thenstore (2820) in memory a record of the moving or sliding of the at leastone finger (e.g. register that for example, a slider has been moved, andthe like), and then optionally change the appearance of the elongatedvirtual key or other control region, such as file names, images, iconsand the like, to visually confirm the command action was executed (e.g.move a knob on a slider control). In one embodiment, alternatively oradditionally, the system may also give an auditory signal that theuser's actions have resulted in the actuating of the drag command andassociated result.

In one embodiment, the user may lift two or more fingers. In otherwords, the system may determine using the model, that a multitude offingers of the user are positioned above but not touching the controlregion of the touchpad. The user may then lower the two or more fingersto the touchpad. In other words, the system may determine that themultitude of fingers are subsequently touching the control region inaccordance with the data and the model.

Then in one embodiment, the system may determine a motion of a firstfinger in relation to a motion of a second finger different than thefirst finger and assign a command to control the computerized system inaccordance with the determined motion. For example the user may eithermove the two fingers further apart on the touchpad to change thedisplayed image, e.g. magnify or zoom-in on a displayed image, move thetwo fingers closer together to zoom-out, or rotate the fingers around arotation point intermediate between the two fingers to rotate an image.The system may be configured to do corresponding display screenoperations under touchpad control where the image on the display screenexpands or contracts, or rotates according to the rotation direction ofthe fingers when the relative motions of the two fingers are assigned tothe respective commands. In one embodiment, the system may not requirethe use of a virtual key or other control regions. Instead, the systemmay operate as if the entire screen is a control region that may besubject to zoom-in, zoom-out, and/or rotation controlled as describedabove.

Most existing two dimensional multi-touch gestures may be similarlyextended or modified into corresponding three-dimensional counterpartsthat incorporate the finger lift gesture component described above.Examples of existing multi-touch gestures that may be modified foradditional finger lift functionality include various Apple OXS gestures,such as, but not limited to: swipe behind full-screen apps, two-fingerscroll, tap to zoom, pinch to zoom, swipe to navigate, open launch pad,show desktop, look up, app expose, rotate, three-finger drag, tap toclick, secondary click, notification center, and show web browser tabs.

The embodiments of the present invention may be extended or modified foruse with touchpads that are capable of sensing the force exerted by afinger in contact with or touching the touchpad. One example of a forcesensing touchpad embodying the present invention may use a touch sensingdevice commercially available from Synaptics Inc., San Jose, Calif. andcommonly known as the ForcePad™ family of products. With a force-sensingtouchpad, the touchpad not only may determine a finger touch location onthe surface of the touchpad, but also may sense and determine how muchforce per finger is being applied to the surface of the touchpad. In oneembodiment, the dimension of force per finger in the touchpad data maybe used by the system instead of or in addition to sensing when a fingerof the user is lifted off the surface of the touchpad.

FIG. 29 depicts a simplified exemplary flowchart of a “lift and tap”technique of key entry modified to use force applied per finger forcontrolling an input from a user to the computerized system, inaccordance with one embodiment of the present invention. FIG. 29includes the same features as FIG. 26 with the following exceptions.When the system obtains (2910) data from the touchpad, the data may bebe additionally associated with the force of a finger and/or hand of theuser upon the touchpad.

Referring to FIGS. 27A-27F in one embodiment, the system may display agraphical representation of at least one touch point associated with theforce per finger (2743, 2750, 2753, 2765, 2770, 2775) applied upon thesurface of the touchpad by the user at the corresponding location on thetouchpad where the fingers touch the surface of the touchpad. In oneembodiment, the graphical representation of the touch point may bedepicted as a solid circle including a diameter associated with theamount of force per finger applied upon the surface of the touchpad. Inalternative embodiments, shapes other than a solid circle may be usedand/or other display attributes than size may be associated with theforce per finger. In one embodiment, the force per finger may beassociated with at least one of a size, a color, a position, a shape, ora display type depicted on the display screen. For example, the amountof force per finger may be associated with a flashing type display wherethe rate of flashing may be associated with the amount of force.

Referring simultaneously to FIG. 27A, FIG. 27D, and FIG. 29, afteranalyzing (2630) the data in accordance with a model of a human hand,the system may optionally determine (2940) that at least one finger isinitially touching the control region using force within a range “A” inaccordance with the data and the model. In one embodiment, force range Amay include a range of force per finger above about 0.5 newton or about50 gram weight equivalent units (on the surface of the Earth),corresponding to when the user touches the surface of the touchpad bypressing firmly. In one embodiment, the system may optionally displaythe graphical representation of the at least one touch point associatedwith the force per finger (2705, 2765) corresponding to force range Adepicted as a solid circle having a relatively large diameter close tothe pitch between adjacent virtual control surfaces, i.e. virtualkeyboard keys, and/or close to the width of user fingers (F2, F3),respectively.

In one embodiment, an audible signal may be generated in addition to orinstead of the graphical representation of the touch point when theuser's finger is initially touching the first control region using forcerange A. In one embodiment, an mechanical response may be generated bythe system in addition to or instead of the graphical representation ofthe touch point when the user's finger is initially touching the firstcontrol region using force range A. In other words, the computerizedsystem may generate a haptic feedback response from the computerizedsystem to the user when the at least one finger of the user is touchingthe first control region using the first force.

For example, the mechanical response may include a haptic response suchas mechanically shaking or vibrating a portion of the computerizedsystem using a mechanical actuator such that the touchpad is shaken bybeing mechanically coupled to the shaken portion of the computerizedsystem. In one embodiment, the shaken touchpad may provide hapticfeedback to the user's fingers in any combination of the visual, i.e.graphical representation, and/or audible feedback to the user indicatingthe user's action has been registered by the computerized system. Inanother embodiment, a different portion of the computerized system thanthe touchpad may be mechanically shaken or vibrated such a portion ofthe computerized system in mechanical contact with the user, e.g. awearable device, or a device supporting the user such as a chair, seat,backrest, elbow rest and/or the like. Haptic feedback may be useful whenaudible feedback is undesirable or ineffective, such as in an audiblynoisy environment.

Referring simultaneously to FIG. 27B, FIG. 27E, and FIG. 29, in oneembodiment, the system may determine (2950), using the model, that atleast one finger of the user is touching a control region of thetouchpad using a force range “B” different than force range A. In oneembodiment, a force per finger from force range A may be greater than aforce per finger from force range B. In one embodiment, force range Bmay include a range of force per finger between zero newton and about0.5 newton. In one embodiment, the system may change the display of thegraphical representation of the at least one touch point associated withthe force per finger (2750, 2770) to a different size, color, position,shape, or display type. For example, the touch point associated with theforce per finger (2750, 2770) may be depicted on the display screen as asmaller solid circle corresponding to a lighter touch of the user'sfinger than when the finger was initially touching the control region inFIG. 27A and FIG. 27D.

In one embodiment, an audible signal may be generated in addition to orinstead of the graphical representation of the touch point when theuser's finger is touching the control region using force range B. In oneembodiment, an mechanical response may be generated by the system inaddition to or instead of the graphical representation of the touchpoint when the user's finger is touching the control region using forcerange B. In other words, the computerized system may generate a hapticfeedback response from the computerized system to the user when theuser's finger is touching the control region using force range B.

In the example depicted in FIG. 27B and FIG. 27E, the finger is still incontact with the surface of the touchpad but with a lighter touch thanthe initial touch. However, the users lighter touch action may beinterpreted by the system in similar fashion as to the earlierembodiments which described the user's finger being lifted completelyoff the touchpad surface, i.e. force equal to zero newton, the lift inthe present embodiment include lifting the finger merely to reduce theforce exerted by the finger but not completely lifting the finger offthe surface of the touchpad. The lighter force lift technique may takeless computational resources, provide faster system speed, and/or betterreliability than when the finger is lifted completely off the touchpadsurface because the lighter force touch point location of the user'sfinger is directly available without having to estimate the position ofa finger lifted completely off the touchpad surface.

Referring simultaneously to FIG. 27C, FIG. 27F, and FIG. 29, in oneembodiment, the system may determine that the at least one finger issubsequently touching the control region using force range A inaccordance with the data and the model. In one embodiment, the systemmay change the display of the graphical representation of the at leastone touch point associated with the force per finger (2753, 2775) to adifferent size, color, position, shape, or display type. For example,the at least one touch point associated with the force per finger (2753,2775) may be depicted on the display screen as a larger solid circlecorresponding once again to a more forceful touch of the user's fingerin force range A similar to that when the finger was initially touchingthe control region in FIG. 27A and FIG. 27D.

FIG. 30 depicts a simplified exemplary flowchart of a modified “lift andtap” technique of key entry modified to use a third force applied perfinger for controlling an input from a user to the computerized system,in accordance with one embodiment of the present invention. FIG. 30depicts the same features as FIG. 29, except FIG. 30 depicts the systemmay determine that the at least one finger is subsequently touching(3060) the control region using a force range “C” in accordance with thedata and the model. The third force applied per finger may be within arange of force per finger from force range C. Force range C may includea range of force per finger that is different than both force range Aand force range B. Use of three force ranges A, B, and C may furtherreduce unintended inputs compared to using just two force ranges. In oneembodiment, force range C may include a range of force per finger thatis greater than force range A, which in turn may be greater than therange of force per finger from force range B. For example, thecomputerized system may respond to a touchpad input sequence from afinger of a user that includes a medium force A, followed by a smallforce B, followed by a large force range C similar to what the user mayuse when typing on mechanically actuated keys. It is understood thatother combinations of the three force ranges may be used in sequence toprovide an input to the computerized system.

In one embodiment, an audible signal may be generated in addition to orinstead of the graphical representation of the touch point when theuser's finger is subsequently touching the first control region usingthe force range A. In one embodiment, an mechanical response may begenerated by the system in addition to or instead of the graphicalrepresentation of the touch point when the user's finger is subsequentlytouching the first control region using the force range A. In otherwords, the computerized system may generate a haptic feedback responsefrom the computerized system to the user when the user's finger issubsequently touching the first control region using the force range A.

The sequence of steps for a user's finger actuating a command area onthe touchpad in the embodiments above included the system responding toan optionally stronger force range, followed by a weaker force range,followed by a stronger force range, in that order. It is understoodthat, in an alternative embodiment, the inverse sequence of force by theuser's finger may be used where the system responds to a weak forcerange (optionally) applied by the user's finger, followed by a strongerforce range, followed by a weaker force range, in that order. In eitheralternative embodiment, the system may recognize and respond to anysequence of a first force range followed by a second force range that isdifferent from the first force range applied by the user's finger toactuate a command area on the touchpad.

The embodiments of the present invention may be extended or modified foruse with not only force-sensing touchpads that may directly determinethe force exerted by a finger in contact with or touching the touchpad,but also with capacitive sensing touchpads. In one embodiment of thepresent invention, a capacitive sensing touchpad may indirectlydetermine the force using a contact area included in the data from thetouchpad. In contrast, force-sensing touchpads directly determine theforce without using a contact area included in the data from thetouchpad.

FIGS. 31A-31B respectively depict simplified exemplary side and topviews of a portion of the touchpad (3110) using the contact area (3120)resulting from a first force FA, in accordance with one embodiment ofthe present invention. FIGS. 31A depicts a user's finger (3130) pressingon the touchpad with a force FA using force range A, in accordance withone embodiment of the present invention as described above. Force rangeA deforms the soft tissue of user's finger (3130) between the user'sbone and the touchpad surface, which is more rigid than the soft tissue,forming a contact area (3120) at the touch point on the touchpad.

FIGS. 32A-32B respectively depict simplified exemplary side and topviews of a portion of the touchpad (3110) using the contact area (3220)resulting from a second force FB, in accordance with one embodiment ofthe present invention. FIGS. 32A depicts user's finger (3130) pressingon the touchpad with a force FB using force range B, which is differentthan force FA, in accordance with one embodiment of the presentinvention as described above. In one embodiment, force range B may beless than force range A. Therefore, force FB deforms the soft tissue ofuser's finger (3130) between the user's bone and the touchpad surface toa lesser extent than when force FA is applied, forming a contact area(3220) at the touch point on the touchpad that has smaller area thancontact area (3120) as depicted respectively in FIGS. 32B and 31B.

The system may use the contact area data from the touchpad to thenindirectly calculate or determine the force range applied by the fingerand/or hand against the touchpad. The contact area information requiresthe soft and/or resilient tissue of the hand to be in contact with thetouchpad without the touchpad supplying the force data directly, whichfor example, precludes the use of a rigid stylus to enter the touchpaddata instead of a user's hand. Once the system calculates or determinesthe finger force range applied to the touchpad, the system may then usethe calculated force information in the same embodiments described inreference to FIGS. 27A-30.

Distinguishing Between Control Regions and “Holding Regions.”

Although users may, for example, use back mounted touchpads to controltheir various handheld computerized devices, in some situations, usersmay simply wish to hold their handheld computerized devices in the sameregion as the back mounted touchpad, which may create false commandinputs when the user inadvertently touches a control region but reallyintends to merely hold the device by touching the touchpad. In these andrelated situations, according to one embodiment, the user may designatea portion of the touchpad surface area as being reserved for non-controlpurposes, e.g. “holding” purposes, hereinafter also referred to as a“non-control” region of the touchpad. In other words, the system enablesthe user to designate or lock out a portion of the touchpad temporarilyas a non-control region for holding the handheld computerized devicewithout controlling an input when the user touches the non-controlregion.

In one embodiment, the system enables the user to designate some or allof a touchpad as being at least temporarily a non-control region or offlimits from a device control perspective by including an actuatingbutton—either real or virtual. Alternatively, in another embodiment,certain user hand gestures, such as a swipe border gesture followed by aswipe “x” gesture within the border, may be assigned and recognized bythe system as temporarily turning off touch control within the portionof the touchpad covered by the border and the swiped “x”. The user maythen safely hold the handheld computerized device or other device by thenon-control region of the touchpad. When the user wishes to return tocontrolling the computerized device using the non-control regions of thetouchpad, the user may then, in one embodiment, actuate a corresponding“restore” (real or virtual) button, or implement an appropriate “restorecontrol” gesture or set of gestures designated to execute the restorecontrol command.

“Push and Lift” or “Enter and Lift” Key Actuation.

FIG. 35 depicts a simplified exemplary flowchart of a “push and lift”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention. FIG. 33 has the same features as FIG. 26 with the followingexceptions. FIG. 33 depicts that the the computerized system firstobtains (3310) data from the touchpad. The data may be associated withthe location and timed movement of a finger and/or hand of the user andnot associated with an image of the finger of the user from an imagesensor, when the user operates the computerized system using thetouchpad. After analyzing (2630) the data in accordance with a model ofa human hand, the computerized system determines that at least onefinger is initially touching a first control region on the touchpad inaccordance with the data and the model. In other words, the user mayinitially slide, touch, or tap one or more fingers over a key area orcontrol region on the touchpad. The system may use the biomechanical andanatomical model of the human hand to recognize this sliding, touching,or tapping action by the user's finger on the touchpad.

In one embodiment, the computerized system may respond by, in turn,changing the visual appearance of the key or control area on the displayas described above. Then, the computerized system determines (3350),using the model, that at least one finger of the user is positionedabove but not touching a first control region of the touchpad during apredetermined time window. In other words, the key or control region onthe touchpad becomes activated or actuated when the system determines,again using the biomechanical and anatomical model of the human hand,that the user has then subsequently lifted their fingers or otherportion of their hand from the key or control region of the touchpad.

FIG. 34 depicts a simplified exemplary time-line of the “push and lift”technique depicted in FIG. 33, in accordance with one embodiment of thepresent invention. The computerized system may use a predefined delaytime Td and a predefined time window Tw to provide the system atechnique to distinguish between deliberate user control actions, andrandom signals that might be caused by the user simply holding thedevice by the input surface area of the touchpad without intending tosend control signals. Referring simultaneously to FIGS. 33-34, thecomputerized system may be configured so that when the system determines(3340) the user initially slides, touches, or taps their fingers orother portions of the hand on a key or control region at time T₀, thereis then only a limited predefined time window, Tw, during or in whichthe system may determine (3350) that a subsequent lifting of the user'sfingers or other portion of their hand is considered to be a deliberatecontrol signal so as to store a record of the lifting of the user'sfingers in memory as described in the embodiments previously described.

In one embodiment, time window Tw may initially commence or open at afirst delay time Td after the system determines (3340) the initialsliding, touching, or tapping motion of the user's finger or hand overthe key or control region on the touchpad at time T₀. Once open, timewindow Tw may then remain open until a time equal to T₀+Td+Tw when thetime window then closes. In other words, the predetermined time windowcloses at the sum of the first delay time and window duration time Tw.Once time window Tw closes or after time T₀+Td+Tw, then if the systemdetermines a finger of the user positioned above but not touching thefirst control region of the touchpad, then the computerized system maynot consider such “finger lift” as a deliberate control signal and maynot store a record of such a finger lift.

In one embodiment, the time duration for delay time Td may be betweenabout 0 and 1 seconds after sliding was first detected at T₀, or inother words, predetermined time window Tw commences after determiningthat the at least one finger is initially touching the first controlregion, which is at T₀. In one embodiment, predetermined time window Twcommences at delay time Td after determining that the at least onefinger is initially touching the first control region. In oneembodiment, delay time Td includes a range of time equal to or less than1 second. In one embodiment, the time duration for window duration timeTw may include a range of time between about 0.25 second to 30 seconds.

In alternative embodiments, other values for delay time Td and windowduration time Tw may also be possible so long as Td and Tw are chosen soas to enable the computerized system to differentiate between fingerlifts intended as inputs versus finger lifts resulting in unintendedinputs, such as for example, during finger repositioning merely tobetter grip or support the touchpad. In other words, determining (3350),using the model, using predetermined time window Tw enables thecomputerized system to differentiate between an intended input versus anunintended input by the at least one finger when the at least one fingeris positioned above but not touching the first control region of thetouchpad and when predetermined time window Tw is not open, such asbefore time T₀+Td and/or after time T₀+Td+Tw. The embodiments describedin reference to FIGS. 33-34 may be combined in any combination with theembodiments referenced in FIGS. 1-32B described previously.

“Hover and Tap” Type Key Activation.

FIG. 35 depicts a simplified exemplary flowchart of a “hover and tap”technique of key entry for controlling an input from a user to thecomputerized system, in accordance with one embodiment of the presentinvention. FIG. 35 depicts features similar to FIG. 26 with thefollowing exceptions. The computerized system obtains (3510) data fromthe touchpad. The data may be associated with the location and movementof a multitude of fingers and/or a hand of the user. But the data is notassociated with an image of a first finger of the multitude of fingersfrom an image sensor, when the user operates the computerized systemusing the touchpad.

FIGS. 36A-36F depict a series of simplified exemplary display screenshots of the “hover and tap” technique of key entry depicted in FIG. 35being used to type and enter the numeral “0” on a prototype computerizedsystem, in accordance with embodiments of the present invention.Referring simultaneously to FIGS. 35 and 36A, after analyzing (2630) thedata in accordance with the biomechanical and anatomical model of thehuman hand, in one embodiment the computerized system may optionallydetermine (3540) that first finger F1 is initially touching a controlregion on the touchpad, in accordance with the data and the model. Forexample, the system displays a graphical representation of virtualtouch-point (3610) of finger F1, touching between virtual keys “9”and/or “0” (3620, 3630) respectively. In one embodiment, thecomputerized system may generate a special graphical representation ofthe control region when first finger F1 is initially touching a controlregion on the touchpad. However, in the embodiments depicted in FIG. 36Abecause touch-point (3610) of finger F1 is in an intermediate positionbetween keys, the keys are still displayed normally. The graphicalrepresentation of finger 1 is near both the number keys “9” and “0” onthe virtual keyboard, but is not yet hovering over either number key, soneither number key is highlighted.

Referring simultaneously to FIGS. 35 and 36B, in one embodiment, thecomputerized system may determine (3550), using the model and the datafrom the touchpad, that finger F1 of the user is positioned above butnot touching, hereinafter also referred to as “hovering”, a controlregion such as virtual key “9” of the touchpad, which in one embodimentmay in turn cause the system to display the control region or virtualkey “9” in a highlighted fashion as described previously. Thebiomechanical and anatomical model of the human hand may be used todetermine when a user's finger is likely hovering above a key or othercontrol area, which when detected may cause the computerized system tohighlight the graphical representation of the displayed control area forvirtual key “9” (3625).

The computerized system may predict (3545) the location of finger F1 inaccordance with the analyzed data and the location of at least anotherfinger of the multitude of fingers, such as for example finger F2different than finger F1 and/or hand (2701). The touch-point data fromfinger F2 and/or hand (2701), which is touching the touchpad, is usedwith the biomechanical and anatomical model of the human hand by thecomputerized system to predict the location of finger F1 even whendirect real-time finger F1 touch-point location data is absent becausefinger F1 is hovering above the touchpad without making touch contactwith the touchpad. For example, it is noted that the position of fingerF2 fully obscures the display of the numeral “6” on virtual key “6”(3630), while the position of finger F3 only partially obscures thedisplay of the numeral “4” on virtual key “4” (3635). In someembodiments, the prediction (3545) step may precede determining when afinger hovers step (3550), while in alternative embodiments theprediction (3545) step may come after determining when a finger hoversstep (3550)

Referring to FIG. 36C, because the user does not wish to type the number“9”, the user continues to move his Finger 1 hover location still abovebut not touching the touchpad towards the location of control region key“0”. It is noted that finger F2 and F3 are still touching the touchpadbut at slightly different positions than in FIG. 36B. Now in FIG. 36Cthe position of finger F2 only partially obscures the display of thenumeral “6” on virtual key “6” (3650) because finger F2 has movedslightly upwards with respect to the figure. Similarly, the position offinger F3 has moved slightly upwards to fully obscures the display ofthe numeral “4” on virtual key “4” (3655). The changed positions of thetouch-points of fingers F2 and F3 are used to predict the new real timeposition of still hovering finger F1. Thus the computerized system intransition briefly enlarges or highlights the displayed graphicalrepresentation of both the previous key “9” (3640) and key “0” (3645).

Referring to FIG. 36D, the user continues to move his Finger 1 so thatit is now hovering over the desired control area of key “0” on thetouchpad, and the computerized system continues to predict the real timelocation of still hovering finger F1 using for example the new real timelocations of fingers F2 and F3, which are still touching the touchpad atlocations slightly different than those depicted in FIG. 36C. Then inone embodiment, the graphical representation of key “0” (3660) isenlarged or otherwise highlighted by the system.

Referring simultaneously to FIGS. 35 and 36E, in one embodiment, thecomputerized system determines (3560) that finger F1 is subsequentlytouching the control region key “0” on the touchpad in accordance withthe data and the model. The user is now actuating the desired key “0” bytouching the corresponding “key 0” region of the touch pad. Note thatthe system is indicating a virtual key “strike” by altering theappearance of the graphical representation as a enlarged or highlightedregion of virtual key “0” 3665, and that additionally the system nowregisters or stores in memory and displays the number “0” (3670) on thetop of the screen. Referring to FIG. 36E, the action of tapping orpressing the control region of key “0” action is now fully complete andthe graphical representation of virtual key “0” (3675) is displayed asat the start in FIG. 36A in original state.

The user actuating a key or other control area by hovering and thenpressing or tapping a key or control area is, from a human factorsstandpoint, easy for the user to learn and use because the user needmerely tap his finger on a hover-highlighted key or control area inorder to then actuate the corresponding virtual key or control area. Oneadvantage of this approach is that that the user does not have to do aseparate tap and lift motion to inform the system about his interest inactuating a particular key. Rather, when the user's finger hovers abovea given virtual key, but does not actually touch it, the system willenlarge or otherwise highlight the key by predicting the location of theuser's hovering finger without having to first touch the touchpad at thedesired control region location. As a result, the hovering finger canmore easily “hover-slide” onto a particular virtual key of interest, asdepicted by the user both hovering and sliding between the key “9” andthe key “0” locations in FIGS. 36B-36D. Thus the user may experience amore natural keyboard typing experience, although the motions andpositions of the fingers are otherwise different since the touchpad maybe obscured from direct user view. The embodiments described inreference to FIGS. 35-36F may be combined in any combination with theembodiments referenced in FIGS. 1-34 described previously.

Tracking a Position of a Portion of the User'S Hand and/or Finger Usinga Virtual Keyboard

In some embodiments, the position and orientation of the virtualkeyboard can be set to track the position and orientation of a portionof the user's hand and/or finger as the user operates the touchpad. Inthis situation, although the virtual keyboard generation software mayoptionally still perform a certain amount of time averaging of theposition and orientation of the portion of the user's hand and fingers,this time averaging may be relatively short, on the average of less thana tenth of a second to a few seconds, or even not at all, in someexamples. Making the time averaging either very short or zero enablesthe system to essentially annotate the image of the moving virtual handand fingers with corresponding virtual keys displayed in a displayscreen of the computerized system. In an embodiment, and as will bediscussed in detail below, these keys may be assigned to the differentfingers of the user in accordance with a standard QUERTY format, orother keyboard formats.

In some embodiments, when the user moves his or her hand around thetouch pad, a portion of the virtual hand and/or the virtual finger maybe within immediate striking range of the various virtual keys, thusenabling the user to start typing immediately regardless of theorientation of the user's hand relative to the touchpad or the devicedisplay screen.

Controlling a Control Region on a Display Screen of a ComputerizedSystem

In accordance with at least some embodiments, the computerized systemmay assign touch data pertaining to a position of a portion of the handof the user when the user operates the touchpad to at least one of amultitude of fingers of the model of the human hand and compute agraphical representation of the user's fingers in accordance with themodel. In some embodiments, the computerized system may be configured toidentify a set of virtual keys to be associated with each of the user'svirtual (i.e., graphical representation of) fingers. The assignment ofthe set of virtual keys to a user's virtual finger may be performed invarious ways. In one example, each virtual finger may be associated witha pre-defined set of keys. For example, each virtual finger may beassociated with a group of keys that the user is familiar with, such asbased on a QUERTY keyboard configuration. Here, for example, the lefthand index finger may be associated with the keys ‘R’, ‘F’, ‘V’, ‘T’,‘G’, ‘B’, which may be the same set of keys that this finger would beexpected to strike if the finger were operating a real QUERTY keyboard.Similarly the right hand middle finger may be associated with the keys‘I’, and ‘K’ and other QUERTY keys that the right hand middle fingerwould be expected to strike if this finger were operating a real QUERTYkeyboard. While the above discussion relates to mapping keys arranged ina QUERTY configuration to the different fingers of the user, it shouldbe appreciated that other finger-key mapping schemes may also beimplemented, in other embodiments. For example, other methods to assignkeys to specific fingers of the user's virtual hand may be used such asnumeric keypad schemes, alternative non-QUERTY keyboard schemes, such asDvorak, Colemak, as well as various variants of the QUERTY keyboardtechnique.

In some embodiments, the computerized system may be configured togenerate a control region comprising the set of virtual keys and displaythe control region in a first position of the display screen of thecomputerized system. In an embodiment, the control region may displaythe set of virtual keys associated with the fingers of the user. When auser's finger that is in contact with the touchpad moves, thecomputerized system may be configured to update the position of the setof virtual keys associated with the movement of that particular user'sfinger on the touchpad, on the display screen. In an alternateembodiment, the control region may also display a graphicalrepresentation of the user's fingers and the different sets of virtualkeys associated with each finger, on the display screen. In such anembodiment, when the user's finger that is in contact with the touchpadmoves, the computerized system may be configured to update the positionof the virtual finger, as well as the group of virtual keys associatedwith the virtual finger in the control region in accordance with themovement of the user's finger on the touchpad.

As an example, using a QUERTY keyboard arrangement as discussed above,if the user's right index finger moves on the touchpad, the virtual keys‘U’, ‘J’, ‘M’ as well as ‘Y’, ‘H’, and ‘N’ associated with the user'sright index finger displayed in the control region may also move. Insome examples, the user's right index finger may be appropriatelypositioned in the control region to strike these keys.

In some examples, resting keys (i.e. keys, such as ‘A’, ‘S’, ‘D’, ‘F’(left hand) and ‘J’, ‘K’, ‘L’, and ‘;’ (right hand) that a relaxeduser's hand would normally contact on a QUERTY keyboard when notpressing a key may also move when the fingertip of the user's fingermoves. As described herein, ‘resting keys’ may refer to the set of keyson a QUERTY keyboard that the user's fingers may typically rest on whennot typing. Thus, in some embodiments, the resting keys may move inaddition to the movement of the set of virtual keys that are associatedwith this particular finger that would normally be expected to be struckby the user's finger.

Thus, using the above technique, the computerized system may beconfigured to position the user's virtual fingers and the correspondingsets of virtual keys associated with the virtual fingers to be locatedanywhere in the display screen according to the hardware limitations ofthe touchpad and the display screen.

In some embodiments, when the user lifts his fingertip from the surfaceof the touchpad (i.e., the user's finger is positioned above but nottouching the touchpad), the keys associated with this finger are bothfrozen or fixed into position, as well as set to an ‘enabled’configuration. In an example, the ‘enabled’ keys may be displayed byvisually highlighting these keys on the display screen. For example, the‘enabled’ keys may be highlighted by showing these keys as brighter, ormore enlarged, or in a different color, and so on.

In some embodiments, the touchpad may include pressure sensitive sensorsthat are capable of sensing the pressure of a touch of a portion of thehand of the user on the touchpad. In such an embodiment, thecomputerized system may obtain touch data pertaining to the amount ofpressure applied by the user when the user operates the touchpad. Basedon the obtained touch data, the computerized system may determinewhether to enable a virtual key associated with the user's finger byvisually highlighting the virtual key. For example, the computerizedsystem may ‘enable’ a virtual key associated with the user's virtualfinger in the control region if the obtained touch data pertaining tothe amount of pressure applied by the user indicates that the pressureis within a pre-determined threshold value. In some embodiments, thispre-determined pressure threshold value may be specified by the user ofthe computerized system.

In certain embodiments, when the user places the tip of the liftedfinger back on to an enabled virtual key of an associated group ofenabled virtual keys (i.e., when the user's finger subsequently touchesthe touchpad), then that particular key is selected (i.e. consideredpressed). This feature thus enables a user to move his or her handfreely on the touchpad surface, while still allowing the user to typeusing conventional typing techniques. Additionally, by freezing orfixing virtual key positions when a user's finger lifts the touchpad,the user can precisely strike a virtual key according to the standardrelative position of that key on a standard keyboard. Thus, thedisclosed technique enables the user to leverage off of a user's longstanding ‘muscle memory’ of relative key positions on a standardkeyboard.

FIG. 37 depicts a simplified exemplary flowchart of a method 3700 forcontrolling a control region on a display screen of a computerizedsystem, in accordance with an embodiment of the present invention. Themethod includes obtaining, using the computerized system, first datafrom a touchpad, the first data being associated with a position of aportion of the hand of a user when the user operates the computerizedsystem using the touchpad at 3702. In an example, the first data is notassociated with an image of a finger of the user from an image sensor.The method then includes transmitting the first data from the touchpadto the computerized device at 3704. In an embodiment, the touchpad maybe located in a location that is different from the location of thedisplay screen. In some examples, at 3706, the method may includeanalyzing the first data in accordance with a model of a human hand andassigning the first data to at least one of a plurality of fingers ofthe model. At 3708, the method may include computing a graphicalrepresentation of a first finger of the portion of the hand of the useron the touchpad in accordance with the model of the human hand.

In certain embodiments, at 3710, the method may include identifying afirst set of virtual keys to be associated with the first finger. At3712, the method may include generating a control region comprising thefirst set of virtual keys. In an embodiment, the control region mayinclude the first set of virtual keys associated with the first fingerof the user. In some embodiments, at 3714, the method may includedisplaying the control region in a first position on the display screenin accordance with the position of the first finger. Additional detailsof the manner in which the method of FIG. 37 may be implemented arediscussed in detail in FIGS. 38 and 39 below.

FIGS. 38A-38F depict a series of simplified exemplary illustrations,3800, 3802, 3804, 3806, 3807 and 3810 of controlling a control regiondisplayed in a display screen of a handheld computerized device by auser operating a touchpad, in accordance with embodiments of the presentinvention. In one embodiment, the control region displays different setsof keys associated with different fingers of the user's hand as the userproceeds to type at touch-typing speed using, for example, the ‘lift andtap’ technique discussed in FIG. 26. It may be noted that at this point,the system has already assigned the touch data from the touchpad to atleast one of the multitude of fingers of the model and has computed agraphical representation of the at least one finger of the user, inaccordance with the model.

Referring now to FIG. 38A, in one embodiment, the system determines (inaccordance with the touchpad data and the model of the human handdiscussed in FIG. 37) that the four fingers (F1, F2, F3, F4) of theuser's right hand 3804 are contacting the touchpad on the back side ofthe computerized device display, but neither the fingers nor the thumbof the user's left hand 3806, nor the thumb of the user's right hand3804, are yet contacting the touchpad. Based on the obtained touchpaddata, the system may then identify different sets of keys to beassociated with each of the virtual fingers (F1, F2, F3, F4) of theuser's hand. In an embodiment, the system may then generate a controlregion comprising a set of virtual keys to be associated with a finger(F1, F2, F3, F4) of the user's hand. In certain embodiments, the systemmay then display the control region in a first position in the displayscreen 3802. In the example depicted in FIG. 38A, the virtual keys (‘Y’,‘H’, ‘N’) and (‘U’, ‘J’, ‘M’) associated with the finger F1 may comprisea first control region. Similarly, the virtual keys (‘I’, ‘K’, ‘,’)associated with the finger F2 may comprise a second control region, thevirtual keys (‘O’, ‘L’, ‘.’) associated with the finger F3 may comprisea third control region and so on. In one example, the letterscorresponding to the sets of keys associated with the fingers (F1, F2,F3, F4) are shown as being positioned near or close to or on top of theuser's fingers.

Referring now to FIG. 38B, the system (in accordance with the touchpaddata and the model of the human hand) may determine that the fourfingers of the user's left hand 3806 are also now touching the touchpadon the back side of the computerized device, and the system thereforegenerates and displays a control region in the display screen 3802comprising the virtual representation of the four fingers of the user'sleft hand and the sets of different keys associated with each of thefour fingers of the user's left hand, on the display screen 3802. In theexample depicted in FIG. 38B, the sets of virtual keys associated witheach hand may be separated and rotated. The example illustrated in FIG.38B further indicates that the user has raised the little finger of theright hand 3804 above the touchpad, causing the letters (‘P’, ‘;’ and‘/’) normally associated with the right little finger to becomehighlighted.

FIG. 38C illustrates all four fingers of each hand of the user touchingthe touchpad. In this situation, system may not highlight any of thekeys associated with the user's fingers and may depict them in theirnormal sizes, thus allowing the user to freely move his hands whileoperating the touchpad. In some situations, the system may not highlighta particular key (e.g., ‘P’) associated with the user's finger if thesystem determines that the user's finger is kept raised (withoutsubsequently touching the touchpad) for a period of time that is greaterthan a pre-determined time out period (e.g., 0-30 seconds) or if theuser's finger touches the outside of the highlighted area associatedwith the particular virtual key.

In FIG. 38D, the user has raised the little finger of the left hand 3806above the keyboard, causing the keys, ‘Q’, ‘A’ and ‘Z’, associated withthis finger to become activated. In the illustrated example, theletters, ‘Q’, ‘A’, ‘Z,’ are shown as activated by highlighting and /orenlarging these letters. In an embodiment, the letters, ‘Q’, ‘A’, ‘Z,’may be fixed (i.e., frozen) into position so that the user can preciselystrike any one of these keys according to the standard relativepositions of those keys on the display screen.

In FIG. 38E, the user has just pressed the location on the touchpad thatcorresponds to the location of the key ‘A’ on the display screen withthe tip of his left little finger. In an embodiment, the system may thenbe configured to depict the newly entered letter ‘A’ at a position(e.g., the top) 3809 of the computerized device screen, as well as startto shrink the enlarged highlighted letters (‘Q’, ‘A’, ‘Z’) since the tipof the left little finger is now in contact with the touchpad again.

In FIG. 38F, since the tip of the left middle finger is again in contactwith the surface of the touchpad, the system no longer highlights thekeys, ‘Q’, ‘A’ and ‘Z’, and depicts them in their normal sizes. Incertain embodiments, the system may be configured to store the letter‘A’ just typed by the user in system memory, as well as shows the letteras being displayed (e.g., 3809) in the display screen 3802.

FIGS. 39A-38F depict a series of simplified exemplary illustrations,3900, 3902, 3904, 3906, 3908 and 3910 of controlling a control regiondisplayed in a display screen of a handheld computerized device by auser operating a touchpad, in accordance with another embodiment of thepresent invention. In the illustrated embodiment, the touchpad isdetached from its normal position on the back of the handheldcomputerized device and positioned in front of the handheld computerizeddevice. It is to be appreciated that by placing the touchpad in front ofthe device, a user may simultaneously observe the position of his handsand fingers, as well as the corresponding motions of his virtual handand fingers and the action of the pressed virtual keys on the displayscreen of the handheld computerized device.

FIG. 39A is an exemplary illustration of a user detaching the touchpad3901 from its normal location on the back of the computerized device toreposition the touchpad to the front of the computerized device. Asdiscussed above, by re-positioning the touchpad to the front of thedevice, the user may simultaneously observe both the actions of theuser's real fingers on the touchpad as well as the corresponding actionsof the user's virtual hand in the display screen.

FIG. 39B is an exemplary illustration of a user touching the detachedtouchpad 3901 near the bottom of the touchpad using the four fingers ofthe user's right hand. In some examples, the user's thumb may notactually touch the touchpad. However, the system, based on the model ofthe user's hand, may determine the position of the user's thumb anddisplay the user's virtual thumb in a corresponding location of thedisplay screen. In an embodiment, the system may be configured todisplay the user's virtual thumb in an alternate color on the displayscreen. The alternate color may enable the user to recognize that thispart of the user's hand is not touching the touchpad. Thus, in someembodiments, the system may be configured to display a more realisticvirtual hand of the user by displaying portions of the user's hand thatare not touching the touchpad. In addition, by displaying portions ofthe user's hand that are not touching the touchpad on the displayscreen, the user's left hand may be visually distinguished from theuser's right hand.

In some embodiments, the system may be configured to assign a set ofvirtual keys to be associated with the user's virtual thumb. In thisimplementation, the user's virtual thumb and its associated virtual keysmay also move in accordance with the user's thumb when the user's thumbtouches and moves around the touchpad. FIG. 39B also displays a controlregion in a first position in the display screen 3903. In the exampledepicted in FIG. 39B, the keys (‘Y’, ‘H’, ‘N’) associated with thefinger F1 may comprise a first virtual region. Similarly, the keys (‘U’,‘J’, ‘M’) associated with the finger F2 may comprise a second controlregion, the keys (‘I’, ‘K’, ‘'’) associated with the finger F3 maycomprise a third control region, and so on. In one example, the letterscorresponding to the sets of keys associated with the fingers (F1, F2,F3, F4) are shown as being positioned near or close to or on top of theuser's fingers.

FIG. 39C is an exemplary illustration of the user moving his left handto a different position of the touchpad. In an embodiment, the systemmay be configured to obtain touchpad data that indicates a position(e.g., a location and/or movement) of a portion of the user's handand/or the user's fingers on the touchpad. Based on the obtained data,in one embodiment, the system may then be configured to determine anangular position of the portion of the user's hand and/or the user'sfingers on the touchpad. The system may then be configured to repositionthe control region comprising the user's virtual fingers and itsassociated virtual keys in accordance with the angular position of theportion of the user's hand and/or the user's fingers on the touchpad.

FIG. 39D is an exemplary illustration of a user not yet raising his leftmiddle finger 3907 above the touchpad. However, since the system has notyet received inputs according to the ‘lift and tap’ technique discussedabove, the system has not yet responded by highlighting the virtualkeys, E′, ‘D’, ‘C’ associated with this finger.

FIG. 39E is an exemplary illustration of the system highlighting (fore.g., by enlarging) the keys ‘E’, ‘D’, ‘C’, normally associated with theuser's left middle finger on the QUERTY keyboard when the system detectsthat the user has raised his left middle finger 3907 above the touchpad.It may be observed that the system has also locked in (i.e., frozen orfixed) the position of these keys so that they now do not move (on thedisplay screen) when the user moves this finger or a portion of his handon the touchpad. FIG. 39E also indicates that the user is in the processof starting to bring the tip of his left middle finger 3907 back ontothe touchpad in the touchpad position corresponding to the letter ‘E’.

In FIG. 39F, since the tip of the left middle finger is again in contactwith the surface of the touchpad, the system no longer highlights thekeys, ‘E’, ‘D’ and ‘C’, and depicts them in their normal sizes. Incertain embodiments, the system may be configured to register aparticular key (e.g., ‘E’) struck by the user and store the letter ‘E’in system memory, as well display the letter ‘E’ in a position 3907 inthe display screen 3903.

Detection of User Gestures

In accordance with at least some embodiments, the computerized system(e.g., the handheld computerized device 100) may be configured to detectan interaction of a portion of the hand of the user when the useroperates the touchpad and/or the display screen of the computerizedsystem. The computerized system may then be configured to cause aproperty of an object displayed on the display screen to be controlledin accordance with the interaction.

In some examples, the object may correspond to a multipage application(e.g., an electronic book), a page oriented application (e.g., a wordprocessing application) or the like, displayed on the display screen ofthe computerized system. For instance, a user may wish to interact withan electronic book displayed on the display screen by turning a page ofthe book. In an embodiment, the computerized system may be configured todetect a finger swipe of the user's finger on the touchpad as an actionindicative of the user's desire to turn a page of the electronic book.In certain embodiments, the computerized system may then be configuredto change the page number of the electronic book in response to thefinger swipe.

In some examples, the computerized system may be configured to detect afinger swipe of a plurality of fingers of the user on the touchpad whenthe user interacts with the multipage application (e.g., an electronicbook) displayed on the display screen. The computerized system may thenbe configured to change the page number by a pre-determined number ofpages in response to the finger swipe. For instance, a single fingerswipe may cause the computerized system to change the page number of themultipage application by a single page, a two finger swipe may cause thecomputerized system to change the page number of the multipageapplication by two pages, a three finger swipe may cause thecomputerized system to change the page number of the multipageapplication by three pages, and the like.

In some embodiments, the computerized system may be configured todetermine the pre-determined number of pages to be changed as a functionof the number of individual fingers used in the finger swipe. As anexample, the pre-determined number of pages to be changed may bedetermined as a function of a power of two raised to the power of thenumber of individual fingers used in each finger swipe, minus 1. As anexample, the pre-determined number of pages to be changed may berepresented by the function: 2^(fingernum−1), where fingernum representsthe number of fingers used in the finger swipe. Using such a function, asingle finger swipe may translate to 2⁰ to change the page number by asingle page, a 2 finger swipe may translate to 2¹ to change the pagenumber by two single pages, a 3 finger swipe may translate to 2² tochange the page number by four pages, a 3 finger swipe may translate to2³ to change the page number by 8 pages and so on. In other embodiments,other base numbers may also be used, such as a base 10 function, forexample, when scrolling thorough extremely large documents. Forinstance, using a base 10 function, a single finger swipe may translateto 10⁰ to change the page number by a single page, a 2 finger swipe maytranslate to 10¹ to change the page number by 10 pages, a 3 finger swipemay translate to 10² to change the page number by 100 pages, a 3 fingerswipe may translate to 10³ to change the page number by 1000 pages, andso on.

In certain embodiments, the computerized system may be configured todetect a velocity of the finger swipe of the user on the touchpad. Thecomputerized system may then be configured to change the page number bya pre-determined number of pages in response to the velocity. Forinstance, the computerized system may be configured to change the pagenumber by a greater amount (e.g., two pages) when it detects arelatively fast finger swipe and change the page number by a smalleramount (e.g., one page) when it detects a relatively slower fingerswipe.

In certain embodiments, the computerized system may be configured todetermine a first direction of the finger swipe and increment the pagenumber by a pre-determined number of pages in response to the firstdirection. For instance, the computerized system may be configured toincrement the page number by a pre-determined number of pages when thecomputerized system detects a finger swipe in the ‘right’ or ‘forward’direction. Similarly, the computerized system may be configured todecrement the page number by a pre-determined number of pages when thecomputerized system detects a finger swipe in the ‘left or ‘backwarddirection.

FIG. 40 depicts simplified exemplary illustrations, 4000 and 4002, thatindicate the manner in which a computerized system (e.g., the handheldcomputerized device 100) may interpret a single finger swipe from a useroperating a touchpad on the back side of the computerized device, inaccordance with an embodiment of the present invention. In theillustrated embodiment, the handheld computerized device 100 may beconfigured to detect a single finger swipe of the user's finger on thetouch pad 200 to control a property (e.g., to change a page number of avirtual page) of an object (e.g., an electronic book) displayed on thedisplay screen 102. In the example shown in 4000, the user's thumb 4004is shown resting on the edge of the device 100 or on the display screen102 while the user's other fingers are shown located behind the device100 and resting on the device's rear touchpad 200 (not shown in FIG.40). The display screen 102 shows a first page, e.g., page 1 from anelectronic book. As further illustrated in the example shown in 4000,the user interacts with the electronic book by extending a finger, 4006,on the touchpad 200 to perform a single finger swipe to turn page 1 ofthe electronic book. The handheld computerized device 100 detects thesingle finger swipe as a command to change the page number to page 2 ofthe book. The device 100 then displays page 2 to the user as shown inthe illustrated example, 4002.

FIG. 41 depicts simplified exemplary illustrations, 4100 and 4102, thatindicate the manner in which a handheld computerized device mayinterpret a multiple finger swipe from a user operating a touchpad onthe back side of the computerized device, in accordance with anotherembodiment of the present invention. In an embodiment, the handheldcomputerized device 100 may be configured to detect a multiple fingerswipe of a plurality of the user's fingers on the touch pad 200 tocontrol a property (e.g., to change the page number by a pre-determinednumber of pages) of an object (e.g., an electronic book) displayed onthe display screen 102. In the example shown in 4100, the user's thumb4104 is shown resting on the edge of the device 100 or on the displayscreen 102 while the user's other fingers are shown located behind thedevice 100, and resting on the device's rear touchpad 200 (not shown inFIG. 41). The display screen 102 shows a first page, e.g., page 1 fromthe electronic book. As further illustrated in the example shown in4100, the user interacts with the book by extending two fingers, 4106and 4108 on the touchpad 200 to perform a two finger swipe across thetouchpad. The handheld computerized device 100 detects the two fingerswipe and interprets the two finger swipe as a command to change thepage number from page 1 to page 3 of the book. The device 100 thendisplays page 3 to the user as shown in the illustrated example, 4102.

In certain embodiments, the computerized system may be configured todetect an interaction of a first finger of the user on a first touchpadof the computerized system with a second finger of the user. In anembodiment, the second finger may be located on a second touchpad of thecomputerized system. The second touchpad may be located in a locationthat is different from the first touchpad. In other embodiments, thesecond finger may also be located on the display screen. In someembodiments, the first touch pad, the second touchpad and the displayscreen may all be located in different locations and need not bephysically connected to each other. For instance, the display screen maybe in nearly any location, such as on a regular monitor, TV screen,projector screen, or on a virtual heads-up eyeglass display worn by theuser (e.g. a device similar to Google Glass).

In certain embodiments, the computerized system may be configured toidentify a first position of the first finger on the first touchpad andidentify a second position of the second finger on the second touchpad.The computerized system may then be configured to detect a selection ofan object by the user. For instance, the object may correspond to afinger controlled device such as a watch dial, a camera, a robotic arm,a two-dimensional (2-D) object, a three-dimensional (3-D) object orother device displayed on the display screen. In some examples, the usermay select the object displayed on the display screen using the firstfinger. In other examples, the user may select the object using thesecond finger on the second touchpad. In certain examples, the user mayalso select the object using both the first finger on the first touchpadas well as the second finger on the second touchpad.

In some embodiments, the computerized system may then be configured todetect a movement of the first position of the first finger relative tothe second position of the second finger and cause a property of anobject displayed on the display screen to be controlled and/or alteredin response to the detected movement. In an alternate embodiment, thecomputerized system may also be configured to detect a movement of thesecond position of the second finger on the second touchpad relative tothe first position of the first finger on the first touchpad and cause aproperty of an object displayed on the display screen to be controlledand/or altered in response to the detected movement. For instance, thecomputerized system may be configured to rotate the displayed objectabout an axis of rotation, change a display characteristic (e.g., color)of the object, alter the size (e.g., enlarge or diminish) of the object,move the object and the like, based on the detected movement.

FIG. 42 depicts exemplary illustrations, 4200, 4202 and 4204 of themanner in which a computerized system (e.g., the hand held computerizedsystem, 100) may detect finger gestures from a user using multiple touchpads and/or the display screen of the computerized system to control anobject displayed on the display screen, in accordance with an embodimentof the present invention. In an embodiment, the user may interact withan object (e.g., 4206) displayed on the display screen 102 by placing afirst finger 4208 on the first touchpad (e.g., 200) located on the backside of the device 100 and a second finger 4210 on the display screen102 (e.g., the front touch screen of the device 100). The user may thenmove the first finger 4208 relative to the second finger 4210 to alter adesired property (e.g., an axis of rotation, a color, a size of theobject and the like). For example, the user may move the first finger4208 relative to the second finger 4210 to rotate the object about anaxis of rotation, change the color of the object, enlarge or diminishthe object, control an angle at which the object is displayed and thelike. In response to the detected movement, the computerized system maythen be configured to cause a particular property (e.g., an axis ofrotation, a color, a size of the object and the like) of the object tobe controlled and/or altered as discussed below.

For instance, in the exemplary illustration shown in 4200, the user'sfirst finger 4208 may be located on the touchpad 200 and the user'ssecond finger may be located on the display screen 102. In this example,the user may move the first finger 4208 in a first direction (e.g.,right) and the second finger 4210 in a second direction (e.g., left). Inthis case, the computerized system may interpret the movement 4212 as atwist about the axis (not shown) of the object and cause the object torotate about its axis.

In the exemplary illustration shown in 4202, the user's first finger4208 and the user's second finger 4210 are located on differenttouchpads or touch surfaces. For example, the user's first finger may belocated on a first touchpad, 4216 and the user's second finger may belocated on a second touchpad, 4218 located in a different location fromthe first touchpad 4216. In other examples, the user's first finger maybe located on the second touchpad 4218 and the user's second finger maybe located on the first touchpad 4216. In the example shown in 4202, theuser may move the first finger 4208 in a downward direction, and asecond finger 4210 in an upward direction. In response to the detectedmovement, the computerized system may cause the object to rotate aboutits axis, 4220.

In the exemplary illustration shown in 4204, the user's first finger4208 is located on the touchpad 200 and the user's second finger 4210 islocated on the display screen (e.g., 102). In this instance, when theuser moves the first finger 4208 in one direction (e.g., left) and thesecond finger 4210 in another direction (e.g., right), the computerizedsystem may interpret this movement as a twist around the object's axis4222 and cause the object to rotate about its axis, 4222.

While the above examples shown in 4200, 4202 and 4204 illustrate arotation of the object about an axis of the object in response to adetected movement and/or interaction between a first finger and a secondfinger of the user, it should be appreciated that the device 100 may beconfigured to cause various other properties of the displayed object tobe controlled in response to a detected movement, in other embodiments.For example, the device 100 may be configured to alter a displaycharacteristic (e.g., color) of the object in response to the detectedmovement. In other examples, the device 100 may be configured to alterthe size (e.g., enlarge or diminish) of the object, move the object andthe like in response to the detected movement.

In certain embodiments, the computerized system may be configured toidentify a first position of the first finger of the user on the firsttouchpad and identify a second position of the second finger on thesecond touchpad. The computerized system may then be configured todetect a selection of a point of rotation of an object displayed on thedisplay screen. For instance, the object may correspond to a virtualjoystick, a virtual hand or mechanical claw displayed on the displayscreen. In some examples, the user may select the object displayed onthe display screen using the first finger. In other examples, the usermay also select the object using the second finger on the secondtouchpad that is located at a different location from the firsttouchpad. In certain examples, the user may also select the object usingboth the first finger on the first touchpad as well as the second fingeron the second touchpad.

The computerized system may then be configured to detect a movement ofthe second finger on the second touchpad relative to the first positionof the first finger on the first touchpad. In an alternate embodiment,the computerized system may also be configured to detect a movement ofthe first position of the first finger on the first touchpad relative tothe second position of the second finger on the second touchpad. Incertain embodiments, the computerized system may then be configured tocause a property of the object to be controlled based on the detectedmovement. For instance, the computerized system may cause the object tomove around the stationary point of rotation based on the detectedmovement. In other examples, the computerized system may push the handleof the virtual joystick in a direction and magnitude defined by acorresponding direction and magnitude of the movement of the secondposition of the second finger relative to the first position of thefirst finger.

FIG. 43 depicts exemplary illustrations, 4300 and 4302 of the manner inwhich a computerized system (e.g., the handheld computerized device 100)may detect finger gestures from a user using multiple touch pads and/orthe display screen of the computerized system to control an objectdisplayed on the display screen, in accordance with an embodiment of thepresent invention. In an embodiment, the user may interact with anobject (e.g., virtual joystick 4312) displayed on the display screen 102by placing a first finger 4304 in a first position 4308 on the touchpad200 located on the back side of the device 100 and a second finger 4306in a second position 4310 on the display screen 102 of the device 100.In an embodiment, the user may hold the first finger 4304 stationary,while moving the second finger 4306 relative to the first finger toalter a desired property (e.g., handle 4314) of the displayed object(i.e., virtual joystick 4312. However, in an alternate embodiment, theuser may move the first finger relative to the second finger whileholding the second finger stationary to alter the desired property(e.g., handle 4314) of the displayed object (i.e., virtual joystick4312). For instance, the user may move the moving finger (i.e., thefirst finger or the second finger) in an upward, downward, left, rightor circular direction 4316 around the first position or the secondposition to alter the desired property (e.g., point of rotation 4316 ofthe handle 4314) of the displayed object (i.e., virtual joystick 4312).In certain embodiments, the computerized system may be configured topush the handle 4314 of the virtual joystick 4312 in the direction andmagnitude as defined by the direction and magnitude of the movement(e.g., 4316) of the first finger in relation to the second finger.

In the exemplary illustration shown in 4302, the computerized system maybe configured to select a stationary point of rotation (i.e. center4316) of the virtual joystick 4312 and its handle 4314 based on a secondposition of the user's second finger 4306 on the display screen 102. Thecomputerized device may further be configured to interpret the movementof the user's first finger 4304 to issue a command to the virtualjoystick 4312 to control the operation of the virtual joystick. (e.g.,to move the virtual joystick's handle 4314 around the joystick handle'sstationary point of rotation 4316)

In accordance with at least some embodiments, the computerized systemmay be configured to identify a first contact position of the firstfinger of the user and detect a selection of an object displayed on thedisplay screen using the first finger. In some examples, the firstfinger may be located on the touchpad 200 located on the back of thedevice 100. In some embodiments, the computerized system may then beconfigured to detect a change in a characteristic of the first contactposition and cause at least one property of the object to be controlledbased on the change in the characteristic of the first contact position.In some examples, the characteristic of the first contact position mayinclude an area, size or shape of the contact position. As an example,the computerized system may be configured to detect a movement of thefirst finger away from the first contact position, a change in the angleof the first finger in the first contact position, a change (increase ordecrease) in a touch area of the first contact position. In response tothe detected movement, the computerized system may then be configured toapply corresponding pressure and/or a corresponding momentum to thedisplayed object.

In certain embodiments, the computerized system may be configured toidentify a first contact position of the first finger of the user on afirst touchpad and a second contact position of a second finger of theuser on a second touchpad, located in a location that is different fromthe first touchpad. The computerized system may then be configured todetect a selection of an object displayed on the display screen usingboth the first finger and the second finger and detect a change in acharacteristic of the first contact position and the second contactposition. In some embodiments, the computerized system may then beconfigured to cause at least one property of the object to be controlledbased on the change in the characteristic of the first contact positionand the second contact position.

FIG. 44 depicts exemplary illustrations, 4400, 4402 and 4404 of themanner in which a computerized system (e.g., the hand held computerizedsystem, 100) may detect finger gestures from a user using multiple touchpads and/or the display screen of the computerized system to control anobject displayed on the display screen, in accordance with an embodimentof the present invention. In an embodiment, the user may interact withan object (e.g., virtual bow and arrow 4418 or a virtual squeeze ball4420) displayed on the display screen 102 by placing a first finger 4406on the touchpad 200 located on the back side of the device 100. Thecomputerized system may then be configured to detect a selection of theobject (e.g., 4418 or 4420) by the user's finger on the touchpad andidentify a first contact area 4408 of the first finger 4406. In someembodiments, the computerized system may then be configured to detect achange in a characteristic of the contact area 4408 and cause a propertyof the displayed object to be controlled based on the detected change inthe contact area.

For instance, as shown in the examples, 4400, 4402 and 4404, thecomputerized system may be configured to detect an increase (e.g., 50%,100% or the like) in the contact or touch area of the first contactposition 4408, for example, when the user presses the touchpad. Thecomputerized system may then be configured to determine a correspondingforce and/or pressure or load to be applied to the displayed objectbased on the detected increase in the touch area. In some examples, thecomputerized system may be configured to apply a first, relativelysmaller force 4414 to the string of the virtual bow and arrow 4418 basedon the first contact position 4408 of the user's finger 4406 and asecond, relatively larger force 4416 to the string of the virtual bowand arrow 4418, based on detecting an increase 4410 to the touch area ofthe first contact position 4408. Similarly, the computerized system maybe configured to apply a first, relatively smaller pressure 4420 to thevirtual squeeze ball 4424 based on the first contact position 4408 ofthe user's finger and a second, relatively larger pressure 4422 to thevirtual squeeze ball 4424, based on detecting an increase 4410 to thetouch area of the first contact position 4408. Additionally, in someexamples, the computerized system may be configured to reduce thepressure (4416, 4422) applied to the virtual object (4418, 4424) whenthe system detects a decrease in the touch area 4410 of the firstcontact position. In some examples, the computerized system may also beconfigured to detect a movement of the first finger away from the firstcontact position, a change in the angle of the first finger in the firstcontact position and the like to alter a property of the displayedobject such as a change in the object's momentum, a force applied to theobject or a change to other parameters of the object.

In accordance with at least some embodiments, the computerized systemmay be configured to detect a movement of a first finger of the userfrom a first position to a second position on the first touchpad. Basedon the detected movement, the computerized system may further beconfigured to reposition an object displayed in the display screen andenable, for the user, an interaction with the re-positioned object. Insome examples, the computerized system may be configured to repositionthe object accordance with a direction of the movement of the firstfinger from the first position to the second position. In otherexamples, the computerized system may be configured to reposition theobject in accordance with an amount of movement of the first finger fromthe first position to the second position.

For instance, a user may wish to operate the handheld computerizeddevice using a single hand. In such a situation, the user may use athumb (often in front of the device) and another finger, such as anindex finger (often on the back of the device) to grip or otherwise holdthe device. The remaining fingers of the user may be positioned on theback of the device, and somewhat free to move (for e.g., over thetouchpad 200 located on the back of the handheld device (100)), but maytypically be constrained by the user's hand geometry to reside near thesame edge of the device where the user's thumb and index finger aregripping the device.

In certain situations, the user may not be able to reach an objectdisplayed on the display screen from the side of the device using theuser's three remaining free (non-gripping) fingers. In such a situation,the user may drag or scoop (i.e. reach out, and then pull in) one ormore of the remaining free (non-gripping) fingers to reach the displayedobject. In certain embodiments, the computerized system may beconfigured to detect this dragging or scooping movement of the user'sfinger and/or fingers towards the location of the object displayed onthe display screen. In some embodiments, the computerized system maythen be configured to re-position the object to a location on thedisplay screen that is within a pre-determined distance from theposition of the user's gripping finger.

FIG. 45 depicts exemplary illustrations, 4500 and 4502 of the manner inwhich a computerized system (e.g., the hand held computerized device100) may detect finger gestures of a user from multiple touch padsand/or the display screen of the computerized system to control anobject displayed on the display screen, in accordance with an embodimentof the present invention. In an embodiment, the user may interact withan object (e.g., a set of virtual keys 4504) displayed in a virtualcontrol region on the display screen 102 by gripping the device using athumb 4510 and an index finger 4512 and using the remaining free fingersto interact with the object. For instance, the user may drag or scoop(i.e. reach out, and then pull in) the middle (non-gripping) finger 4506to reach the set of virtual keys 4504. In an embodiment, thecomputerized system may be configured to detect the movement of thefinger 4506 towards the set of virtual keys 4504 and re-position the setof virtual keys 4504 to location that is closer (i.e., is within apredetermined distance) to the index finger 4512 or the thumb 4510 ofthe user on the touchpad. Once the location of the virtual keys isbrought to within striking range of the free (non-gripping) finger 4506,the user may strike the virtual keys 4504 using the free (non-gripping)finger 4506, and thus interact with the object. Additionally, the usermay move the virtual keys 4504 within striking range of the non-grippingfinger 4506 by moving the finger 4506 over the touchpad 200, in adirection 4508 either towards or away from the position of the set ofvirtual keys 4504.

FIG. 46 depicts a simplified exemplary flowchart of a method 4600 forcontrolling a control region on a display screen of a computerizedsystem, in accordance with an embodiment of the present invention. Themethod includes obtaining, using the computerized system, first datafrom a first touchpad, the first data being associated with a positionof a portion of the hand of a user when the user operates thecomputerized system using the first touchpad at 4602. In an example, thefirst data is not associated with an image of a finger of the user froman image sensor. The method then includes transmitting the first datafrom the first touchpad to the computerized device at 4604. In anembodiment, the first touchpad may be located in a location that isdifferent from the location of the display screen. In some examples, at4606, the method may include analyzing the first data in accordance witha model of a human hand and assigning the first data to at least one ofa plurality of fingers of the model. At 4608, the method may includedetecting an interaction of a portion of the hand of the user on thefirst touchpad with the object displayed on the display screen. In someexamples, at 4610, the method may include causing, by the computerizedsystem, at least one property of the object to be controlled inaccordance with the interaction of the portion of the hand of the useron the first touchpad.

In some embodiments, the method may include detecting a finger swipe ofa first finger of the user on the first touchpad when the user interactswith the at least one object on the display screen. In an example, theobject may correspond to a multipage application displayed on thedisplay screen and the property of the object may correspond to a pagenumber in the multipage application. In an embodiment, the method mayinclude changing the page number of the multipage application inresponse to the finger swipe.

In some embodiments, the method may include detecting a finger swipe ofa plurality of fingers of the user on the first touchpad when the userinteracts with the object on the display screen. In an example, themethod may include changing the page number by a pre-determined numberof pages in response to the finger swipe of the plurality of fingers. Insome examples, the method may include detecting a velocity of the fingerswipe and changing and the page number by the pre-determined number ofpages in response to the velocity. In some examples, the method mayinclude determining a first direction of the finger swipe andincrementing the page number by a pre-determined number of pages inresponse to the first direction. In some examples, the method mayinclude determining a second direction of the finger swipe anddecrementing the page number by a pre-determined number of pages inresponse to the second direction. In an example, the first direction maybe different from the second direction.

In accordance with some embodiments, the method may include detecting aninteraction of at least a second finger of the user with the firstfinger. In some examples, the second finger may be located a secondtouchpad. In some examples, the second touchpad may be located in alocation different from the first touchpad. In some examples, the secondfinger may also be located on the display screen. In some embodiments,the method may include identifying a first position of the first fingeron the first touchpad, detecting a selection of the object, identifyinga second position of the second finger on the second touchpad anddetecting a movement of the first position of the first finger relativeto the second position of the second finger. In some examples, themethod may include detecting the selection of the object by the firstfinger on the first touchpad or the second finger on the secondtouchpad. In some embodiments, the method may include rotating theobject based on the detected movement, altering an axis of rotation ofthe object based on the detected movement, altering the size of theobject based on the detected movement, altering a display characteristicof the object based on the detected movement or moving the object basedon the detected movement.

In accordance with at least some embodiments, the method may includeidentifying a first position of the first finger on the display screen,detecting a selection of a point of rotation of the object, identifyinga second position of the second finger on the second touchpad anddetecting a movement of the second position of the second fingerrelative to the first position of the first finger. In some examples,the method may include detecting the selection of the point of rotationof the object by the first finger on the first touchpad or the secondfinger on the second touchpad. In some examples, the method may includemoving the object around the point of rotation of the object based onthe detected movement. In some examples, the object may correspond to avirtual joystick and the method may include pushing the handle of thevirtual joystick in a direction and magnitude defined by a correspondingdirection and magnitude of the movement of the second position of thesecond finger relative to the first position of the first finger.

In accordance with at least some embodiments, the method may includedetecting a selection of the object, identifying a first contactposition of the first finger, detecting a change in a characteristic ofthe first contact position and causing the at least one property of theobject to be controlled based on the change in the characteristic. In anembodiment, the method may include detecting a movement of the firstfinger away from the first contact position, detecting a change in theangle of the first finger in the first contact position, detecting anincrease in a touch area of the first contact position and the like. Inan example, the characteristic may comprise at least one of the area,the size or the shape of the first contact position. In an embodiment,the method may include applying a corresponding pressure or a load tothe displayed object based on the change in the characteristic of thefirst contact position.

In accordance with at least some embodiments, the method may includedetecting a movement of the first finger from a first position to asecond position on the first touchpad, repositioning the object in thedisplay screen in accordance with a direction of the movement of thefirst finger or an amount of movement of the first finger from the firstposition to the second position and enabling, for the user, aninteraction with the re-positioned object. In an example, the object maycorrespond to a set of virtual keys in a virtual control region in thedisplay screen.

The above embodiments of the present invention are illustrative and notlimiting. Various alternatives and equivalents are possible. Although,the invention has been described with reference to a handheldcomputerized device by way of an example, it is understood that theinvention is not limited by the type of computerized device or systemwherever the device or system may benefit by differentiating between auser's touch on a touchpad for command input and a user's touch on atouchpad for merely holding the device by the touchpad. Although, theinvention has been described with reference to certain user fingerstouching the touchpad by way of an example, it is understood that theinvention is not limited by which user fingers are touching thetouchpad. Although, the invention has been described with reference to atouchpad located on the back of a handheld device including a display atthe front of the device by way of an example, it is understood that theinvention is not limited by where the touchpad is located. Although, theinvention has been described with reference to a capacitive touchpadused for data entry by way of an example, it is understood that theinvention is not limited by the type of input device. Although, theinvention has been described with reference to a sequence of strong,weak, strong or medium, small, large force applied by a user's fingerused for data entry by way of examples, it is understood that theinvention is not limited by those two sequences of forces applied. Otheradditions, subtractions, or modifications are obvious in view of thepresent disclosure and are intended to fall within the scope of theappended claims.

Trademarks: iPAD™ and iPhone™ are trademarks of Apple Inc., Cupertino,Calif. Surface™ is a copyright of Microsoft Corporation.

What is claimed is:
 1. A computer-implemented method for controlling acontrol region on a display screen of a computerized system, the methodcomprising: obtaining, using the computerized system, first data from afirst touchpad, the first data being associated with a position of aportion of the hand of a user when the user operates the computerizedsystem using the first touchpad, the first data not being associatedwith an image of a finger of the user from an image sensor; transmittingthe first data from the first touchpad to the computerized system, thefirst touchpad being located in a location that is different from thelocation of the display screen; analyzing the first data in accordancewith a model of a human hand and assigning the first data to at leastone of a plurality of fingers of the model; detecting, by thecomputerized system, an interaction of at least the portion of the handof the user on the first touchpad with at least one object displayed onthe display screen; and causing, by the computerized system, at leastone property of the object to be controlled in accordance with theinteraction of the portion of the hand of the user on the firsttouchpad.
 2. The computer-implemented method of claim 1, whereindetecting an interaction of at least the portion of the hand comprisesdetecting a finger swipe of a first finger of the user on the firsttouchpad when the user interacts with the at least one object on thedisplay screen.
 3. The computer-implemented method of claim 2, whereinthe object corresponds to a multipage application displayed on thedisplay screen and the at least one property of the object correspondsto a page number in the multipage application.
 4. Thecomputer-implemented method of claim 3, wherein causing the at least oneproperty of the object to be controlled comprises changing the pagenumber of the multipage application in response to the finger swipe. 5.The computer-implemented method of claim 1, wherein detecting aninteraction of at least the portion of the hand further comprisesdetecting a finger swipe of a plurality of fingers of the user on thefirst touchpad when the user interacts with the at least one object onthe display screen.
 6. The computer-implemented method of claim 5,wherein causing the at least one property of the object to be controlledcomprises changing the page number by a pre-determined number of pagesin response to the finger swipe of the plurality of fingers.
 7. Thecomputer-implemented method of claim 6, further comprising detecting avelocity of the finger swipe, wherein causing the at least one propertyof the object to be controlled comprises changing the page number by thepre-determined number of pages in response to the velocity.
 8. Thecomputer-implemented method of claim 3, further comprising: determininga first direction of the finger swipe; and incrementing the page numberby a pre-determined number of pages in response to the first direction.9. The computer-implemented method of claim 8, further comprising:determining a second direction of the finger swipe; and decrementing thepage number by a pre-determined number of pages in response to thesecond direction, the second direction being different from the firstdirection.
 10. The computer-implemented method of claim 2, whereindetecting an interaction of at least the portion of the hand of the useron the first touchpad further comprises: detecting an interaction of atleast a second finger of the user with the first finger.
 11. Thecomputer-implemented method of claim 10, wherein the second finger islocated a second touchpad, the second touchpad being located in alocation different from the first touchpad.
 12. The computer-implementedmethod of claim 11, wherein the second finger is located on the displayscreen.
 13. The computer-implemented method of claim 10, furthercomprising: identifying a first position of the first finger on thefirst touchpad; identifying a second position of the second finger onthe second touchpad; detecting a selection of the object; and detectinga movement of the first position of the first finger relative to thesecond position of the second finger.
 14. The computer-implementedmethod of claim 13, further comprising detecting the selection of theobject using the first finger on the first touchpad.
 15. Thecomputer-implemented method of claim 13, further comprising detectingthe selection of the object using the second finger of the user on thesecond touchpad.
 16. The computer-implemented method of claim 13,wherein causing the at least one property of the object to be controlledcomprises rotating the object based on the detected movement.
 17. Thecomputer-implemented method of claim 13, wherein causing the at leastone property of the object to be controlled comprises altering an axisof rotation of the object based on the detected movement.
 18. Thecomputer-implemented method of claim 13, wherein causing the at leastone property of the object to be controlled comprises altering the sizeof the object based on the detected movement.
 19. Thecomputer-implemented method of claim 13, wherein causing the at leastone property of the object to be controlled comprises altering a displaycharacteristic of the object based on the detected movement.
 20. Thecomputer-implemented method of claim 13, wherein causing the at leastone property of the object to be controlled comprises moving the objectbased on the detected movement.
 21. The computer-implemented method ofclaim 10, further comprising: identifying a first position of the firstfinger on the first touchpad identifying a second position of the secondfinger on the second touchpad; detecting a selection of a point ofrotation of the object; and detecting a movement of the second positionof the second finger relative to the first position of the first finger.22. The computer-implemented method of claim 21, further comprisingdetecting the selection of the point of rotation of the object using thefirst finger on the first touchpad.
 23. The computer-implemented methodof claim 21, further comprising detecting the selection of the point ofrotation of the object using the second finger on the second touchpad.24. The computer-implemented method of claim 21, wherein causing the atleast one property of the object to be controlled comprises moving theobject around the point of rotation of the object based on the detectedmovement.
 25. The computer-implemented method of claim 21, wherein theobject comprises a virtual joystick and wherein causing the at least oneproperty of the object to be controlled comprises pushing the handle ofthe virtual joystick in a direction and magnitude defined by acorresponding direction and magnitude of the movement of the secondposition of the second finger relative to the first position of thefirst finger.
 26. The computer-implemented method of claim 1, furthercomprising: identifying a first contact position of a first finger;detecting a selection of the object; detecting a change in acharacteristic of the first contact position; and causing the at leastone property of the object to be controlled based on the change in thecharacteristic.
 27. The computer-implemented method of claim 26, whereindetecting the change in the characteristic of the first contact positioncomprises detecting a movement of the first finger away from the firstcontact position.
 28. The computer-implemented method of claim 26,wherein detecting the change in the characteristic of the first contactposition comprises detecting a change in the angle of the first fingerin the first contact position.
 29. The computer-implemented method ofclaim 26, wherein detecting the change in the characteristic of thefirst contact position comprises detecting a change in a touch area ofthe first contact position.
 30. The computer-implemented method of claim26, wherein the characteristic comprises at least one of the area or thesize of the first contact position.
 31. The computer-implemented methodof claim 26, wherein the characteristic comprises the shape of the firstcontact position.
 32. The computer-implemented method of claim 26,wherein causing the at least one property of the object to be controlledcomprises applying a corresponding pressure to the displayed objectbased on the change in the characteristic of the first contact position.33. The computer-implemented method of claim 26, wherein causing the atleast one property of the object to be controlled comprises applying acorresponding momentum to the displayed object based on the change inthe characteristic of the first contact position.
 34. Thecomputer-implemented method of claim 1, further comprising: detecting amovement of a first finger from a first position to a second position onthe first touchpad; repositioning the object in the display screen inaccordance with a direction of the movement of the first finger from thefirst position to the second position; and enabling, for the user, aninteraction with the re-positioned object.
 35. The computer-implementedmethod of claim 1, wherein detecting an interaction of the first fingerof the user further comprises: detecting a movement of a first fingerfrom a first position to a second position on the first touchpad;repositioning the object in the display screen in accordance with anamount of movement of the first finger from the first position to thesecond position; and enabling, for the user, an interaction with there-positioned object.
 36. The computer-implemented method of claim 1,wherein the object corresponds to a set of virtual keys in a virtualcontrol region in the display screen.
 37. A non-transitorycomputer-readable storage medium comprising instructions which whenexecuted by a computer cause the computer to: obtain first data from afirst touchpad, the first data being associated with a position of aportion of the hand of a user when the user operates the computerizedsystem using the first touchpad, the first data not being associatedwith an image of a finger of the user from an image sensor; transmit thefirst data from the first touchpad to the computerized device, the firsttouchpad being located in a location that is different from the locationof the display screen; analyze the first data in accordance with a modelof a human hand and assigning the first data to at least one of aplurality of fingers of the model; detect an interaction of at least theportion of the hand of the user on the first touchpad with at least oneobject displayed on the display screen; and cause at least one propertyof the object to be controlled in accordance with the interaction of theportion of the hand of the user on the first touchpad.
 38. Thecomputer-readable media of claim 37, wherein the instructions to detectan interaction of at least the portion of the hand comprisesinstructions to detect a finger swipe of a first finger of the user onthe first touchpad when the user interacts with the at least one objecton the display screen.
 39. The computer-readable media of claim 38,wherein the instructions to detect an interaction of at least theportion of the hand of the user on the first touchpad further compriseinstructions to detect an interaction of at least a second finger of theuser with the first finger, wherein the second finger is located asecond touchpad, the second touchpad being located in a locationdifferent from the first touchpad.
 40. The computer-readable media ofclaim 39, further comprising instructions to: identify a first positionof the first finger on the first touchpad; identify a second position ofthe second finger on the second touchpad; detect a selection of theobject; and detect a movement of the first position of the first fingerrelative to the second position of the second finger.
 41. Thecomputer-readable media of claim 40, wherein the instructions to causethe least one property of the object to be controlled compriseinstructions to rotate the object based on the detected movement. 42.The computer-readable media of claim 40, wherein the instructions tocause the least one property of the object to be controlled compriseinstructions to alter the size of the object based on the detectedmovement.
 43. The computer-readable media of claim 40, wherein theinstructions to cause the least one property of the object to becontrolled comprise instructions to alter a display characteristic ofthe object based on the detected movement.
 44. The computer-readablemedia of claim 37, further comprising instructions to: identify a firstcontact position of a first finger; detect a selection of the object;detect a change in a characteristic of the first contact position; andcause the at least one property of the object to be controlled based onthe change in the characteristic.
 45. The computer-readable media ofclaim 44, wherein the instructions to detect the change in thecharacteristic of the first contact position further compriseinstructions to detect a change in a touch area of the first contactposition.
 46. The computer-readable media of claim 44, wherein thecharacteristic comprises at least one of the area, the size or the shapeof the first contact position.
 47. The computer-readable media of claim44, wherein the instructions to cause the at least one property of theobject to be controlled further comprises instructions to apply acorresponding pressure to the displayed object based on the change inthe characteristic of the first contact position.
 48. Thecomputer-readable media of claim 37, further comprising instructions to:detect a movement of the first finger from a first position to a secondposition on the first touchpad; reposition the object in the displayscreen in accordance with a direction of the movement of the firstfinger from the first position to the second position; and enable, forthe user, an interaction with the re-positioned object.
 49. A system forcontrolling a control region on a display screen of a computerizedsystem configured to: obtain first data from a first touchpad, the firstdata being associated with a position of a portion of the hand of a userwhen the user operates the computerized system using the first touchpad,the first data not being associated with an image of a finger of theuser from an image sensor; transmit the first data from the firsttouchpad to the computerized device, the first touchpad being located ina location that is different from the location of the display screen;analyze the first data in accordance with a model of a human hand andassigning the first data to at least one of a plurality of fingers ofthe model; detect an interaction of at least the portion of the hand ofthe user on the first touchpad with at least one object displayed on thedisplay screen; and cause at least one property of the object to becontrolled in accordance with the interaction of the portion of the handof the user on the first touchpad.
 50. The system of claim 49, furtherconfigured to detect a finger swipe of a first finger of the user on thefirst touchpad when the user interacts with the at least one object onthe display screen.
 51. The system of claim 49, further configured todetect an interaction of at least a second finger of the user with thefirst finger, wherein the second finger is located a second touchpad,the second touchpad being located in a location different from the firsttouchpad.
 52. The system of claim 51 further configured to: identify afirst position of the first finger on the first touchpad; identify asecond position of the second finger on the second touchpad detect aselection of the object; and detect a movement of the first position ofthe first finger relative to the second position of the second finger.53. The system of claim 51 further configured to rotate the object basedon the detected movement.
 54. The system of claim 51 further configuredto: identify a first contact position a first finger; detect a selectionof the object; detect a change in a characteristic of the first contactposition; and cause the at least one property of the object to becontrolled based on the change in the characteristic.
 55. The system ofclaim 51 further configured to apply a corresponding pressure to thedisplayed object based on the change in the characteristic of the firstcontact position.